We need a test set that focuses on adaptor-specific code generation. It should test for specific knowledge contained in different parts of the adaptor documentation across several adaptors.
Example of a user feedback case that needs to be targeted:
"AI will write a sendMessages call to Gmail and include as an attribute « text » or « message » which are undefined (and cause it to crash), instead of using « body » which is clearly defined in the adaptor doc."
This case currently fails because we currently only add adaptor signatures to prompts, while this requires knowledge from the "Interfaces" section of the Gmail docs.
We need a test set that focuses on adaptor-specific code generation. It should test for specific knowledge contained in different parts of the adaptor documentation across several adaptors.
Example of a user feedback case that needs to be targeted:
"AI will write a sendMessages call to Gmail and include as an attribute « text » or « message » which are undefined (and cause it to crash), instead of using « body » which is clearly defined in the adaptor doc."
This case currently fails because we currently only add adaptor signatures to prompts, while this requires knowledge from the "Interfaces" section of the Gmail docs.