More memory-efficient multi-start optimization

[dweindl]

The current multi-start optimization scheme in pypesto.optimize.optimize.minimize relies on creating copies for all N starts upfront. For larger number of starts and large problems, this can become very memory-intensive. It would be great if the problem copies would be created (and destroyed) dynamically when a previous task finished.

Relevant code:

pyPESTO/pypesto/optimize/optimize.py

Lines 145 to 158 in 47ae5d5

	tasks = []
	for startpoint, id in zip(startpoints, ids):
	task = OptimizerTask(
	optimizer=optimizer,
	problem=problem,
	x0=startpoint,
	id=id,
	history_options=history_options,
	optimize_options=options,
	)
	tasks.append(task)
	# perform multistart optimization
	ret = engine.execute(tasks, progress_bar=progress_bar)

CC @stephanmg

EDIT: I think the actual copies are only created by the MultiProcessEngine:

pyPESTO/pypesto/engine/multi_process.py

Line 73 in 47ae5d5

pickled_tasks = [pickle.dumps(task) for task in tasks]

Changing the up-front pickling and Pool.imap to ProcessPoolExecutor.submit with on-demand pickling should already give a good improvement.