Is it possible to use AI to reverse engineer software?

This is the proposed way to reverse engineer software using AI.

• Program fake_user operates program target_prog in diverse ways to generate a huge and comprehensive data set.
• The parameters of an artificial network are trained to produce within specified accuracy and reliability criteria a behavioral equivalent of target_prog.

Not only is this possible, but it is becoming standard practice for AI projects other than reverse engineering games.

There are caveats.

• Program target_prog may be of sufficient complexity to exceed the capabilities of existing network designs and convergence techniques.
• The project may lack access to funds and computing resources to complete the generation and training required to achieve reasonable accuracy, with sufficient reliability, in the time allotted.
• The expertise of those involved may not be sufficient to produce satisfactory results.
• Although the source code is not copied and the parameter state achieved through learning contains equivalent functionality, there is no guarantee that a civil liability may not result. Copyright law one or more jurisdictions may be interpreted as a protection against this kind of copying even though the text of the source code was not copied verbatim.

Remarkably, more or less the scenario you describe is not only feasible and has already been demonstrated (detailed explanation and fascinating videos at link).

However, the fidelity of the copy is currently quite limited:

So for now, your copy will be quite low quality. However, there is a big exception to this rule: if the software you are copying is itself based on machine learning, then you can probably make a high-quality copy quite cheaply and easy, as I and my co-authors explain in this short article.

Interesting question and I'm quite sure that the correct answer will change rapidly over the next few years.