Integration with AllenNLP

[aleSuglia]

Problem

AllenAct represents a framework for learning to act in 3D environments. Learning complex tasks from scratch, without human supervision is very challenging. Additionally, learning from raw pixels is even harder. Language can represent a relevant medium for improving the sample efficiency of learning to act models. Therefore, I believe that a learning to act framework should be also tightly coupled with a learning to understand language framework.

Desired solution

I would like to see a tight integration between AllenNLP and AllenAct. Particularly, the natural language processing component I believe could be essential for performing specific embodied tasks requiring language understanding.

Do you think this is a reasonable feature to consider? If yes, do you have a plan for it? Happy to contribute to this effort :)

[Lucaweihs]

Hi @aleSuglia,

I think this is a fantastic suggestion. One of the reason we haven't prioritized this yet is that the most popular embodied tasks (e.g. instruction following for navigation) don't seem to require extremely deep language understanding. I see you've done some interesting work with guessing games, is there a task you're interested in where you feel like a tight AllenNLP+AllenAct integration would be very helpful?

[aleSuglia]

Hi @Lucaweihs,

Thanks for your answer. The kind of tasks that I have in mind are those in which agents can interact with environment as well as in natural language with other agents to achieve a given task (e.g. https://arxiv.org/abs/2011.08277). I believe that AllenAct combined with AllenNLP would be the perfect solution to tackle this sort of embodied and communicative tasks. I would be more than happy to start drafting a proposal for this with your help!

[aleSuglia]

Any more thoughts on this?