Migrate the PDF build to Buildkite

[DilumAluthge]

See also JuliaCI/julia-buildkite#198

[StefanKarpinski]

Thanks for doing that. Very much look forward to not having someone complain about the broken PDF link for the first time in the past seven years.

[mortenpi]

@DilumAluthge Buildkite is not yet deploying the PDF though, right? Or did I miss something?

[DilumAluthge]

Ah you're right, I forgot we hadn't implemented the upload step yet.

What are the commands that need to be run for the upload? (Assuming that we have the PDF file.)

[mortenpi]

This is the function that handle the actual push to the assets branch:

docs.julialang.org/pdf/make.jl

Lines 140 to 190 in 16a515e

	function commit()
	if get(ENV, "GITHUB_EVENT_NAME", nothing) == "pull_request"
	@info "skipping commit from pull requests."
	return
	end
	if !isdir(JULIA_DOCS_TMP)
	@info "No new PDFs created, skipping commit."
	return
	end
	@info "committing built PDF files."
	# Make sure the repo is up to date
	run(`git fetch origin`)
	run(`git reset --hard origin/assets`)
	# Copy file from JULIA_DOCS_TMP to JULIA_DOCS
	for file in readdir(JULIA_DOCS_TMP)
	endswith(file, ".pdf") || continue
	from = joinpath(JULIA_DOCS_TMP, file)
	@debug "Copying a PDF" file from pwd()
	cp(from, file; force = true)
	end
	mktemp() do keyfile, iokey; mktemp() do sshconfig, iossh
	# Set up keyfile
	write(iokey, base64decode(get(ENV, "DOCUMENTER_KEY_PDF", "")))
	close(iokey)
	chmod(keyfile, 0o600)
	# Set up ssh config file
	print(iossh,
	"""
	Host github.com
	StrictHostKeyChecking no
	HostName github.com
	IdentityFile $keyfile
	BatchMode yes
	""")
	close(iossh)
	chmod(sshconfig, 0o600)
	# Configure git
	run(`git config user.name "docs.julialang.org"`)
	run(`git config user.email "[email protected]"`)
	run(`git remote set-url origin [email protected]:JuliaLang/docs.julialang.org.git`)
	run(`git config core.sshCommand "ssh -F $(sshconfig)"`)
	# Committing all .pdf files
	run(`git add '*.pdf'`)
	run(`git commit --amend --date=now -m "PDF versions of Julia's manual."`)
	# Push
	run(`git push -f origin assets`)
	end end
	end

Basically, add/update the PDFs to the branch, and then commit --amend and force push. We squash the history on every commit so that the assets branch wouldn't blow up in size due to the PDFs from master.

[DilumAluthge]

We also need to rename the PDF file, right? Currently, the PDF file being uploaded as a Buildkite artifact is just named TheJuliaLanguage.pdf. Where is the code/logic for determining the new name of the PDF file?

[mortenpi]

Sorry for the delay. But if I'm not mistaken, Documenter already sets the name, and we just copy any PDF there is: https://github.com/JuliaDocs/Documenter.jl/blob/998d26e568a20e3b1764d0c2cef30ebe4d2563ae/src/Writers/LaTeXWriter.jl#L167-L174

[DilumAluthge]

Hmmm. I'm looking at a recent run on master (https://buildkite.com/julialang/julia-master/builds/17280#0183fada-b0ed-4d00-ab04-1d752f70514d), and in the artifacts tab, the PDF file is just named TheJuliaLanguage.pdf. So it doesn't look like any renaming is happening currently on Buildkite.

[mortenpi]

Oh, sorry, you are right. The renaming logic works by passing this

docs.julialang.org/pdf/make.jl

Line 74 in 16a515e

file = "julia-$(v).pdf"

which has the version number, to this

docs.julialang.org/pdf/make.jl

Lines 58 to 69 in 16a515e

	function copydocs(file)
	isdir(JULIA_DOCS_TMP) || mkpath(JULIA_DOCS_TMP)
	output = "$(JULIA_SOURCE)/doc/_build/pdf/en"
	destination = "$(JULIA_DOCS_TMP)/$(file)"
	for f in readdir(output)
	if startswith(f, "TheJuliaLanguage") && endswith(f, ".pdf")
	cp("$(output)/$(f)", destination; force=true)
	@info "finished, output file copied to $(destination)."
	break
	end
	end
	end

where we look for any PDF and rename it to file. The simplest solution might be to run a small Julia script on Buildkite that checks if it's a tagged Julia, like we do in doc/make.jl to determine if it is a tagged version or not.

Although I guess a more robust solution would be to handle it in doc/make.jl? The Julia docs build will indeed always produce TheJuliaLanguage.pdf right now. We use the .version field for the version, which we could determine from TRAVIS_TAG since we do not explicitly set in the Julia make.jl, but TRAVIS_TAG is not set anywhere. So we will never attach the version number to the filename in Documenter. We could also set the version option to LaTeX there.

[DilumAluthge]

but TRAVIS_TAG is not set anywhere

The Buildkite equivalent is BUILDKITE_TAG.