I've been adding C# code completion functionality to my REPL tool, and ended up reverting to the text-davinci model.
The codex (discontinued?) and text-davinci models gave much better results than GPT3.5-turbo, specifically for code completion scenarios. The latest models seem to produce invalid code, mostly having trouble at the boundaries where they start the completion.
My suspicion is that these latter models focus more on conversation semantics than code completion, and completing code "conversationally" vs completing code in a syntactically valid way has differences.
For example, if the last line of code to be completed is a comment, the model will happily continue to write code on the same line as the comment. Not an issue in a conversation model as there is a natural break in a conversation, but when integrating with tooling it's challenging.
Most likely the issue is that I'm not yet effective at prompt engineering, but I had no issues iterating on prompts for the earlier models. I'm loving the DaVinci model and it's working really well -- I just hope it's not discontinued too soon in favor of later models.
I can corroborate that text-davinci gives much better results than for tasks involving summarization or extraction of key sentences among a large corpus. I wonder what empirical metrics OpenAI uses to determine performance benchmarks for practical tasks like these. You can see the model in action for analysis of reviews here: https://show.nnext.ai/
> it's more hacking than crareful and well specified engineering, and that could lead down a path of instability in the product where some features get better while others get worse, without understanding exactly why.
The codex (discontinued?) and text-davinci models gave much better results than GPT3.5-turbo, specifically for code completion scenarios. The latest models seem to produce invalid code, mostly having trouble at the boundaries where they start the completion.
My suspicion is that these latter models focus more on conversation semantics than code completion, and completing code "conversationally" vs completing code in a syntactically valid way has differences.
For example, if the last line of code to be completed is a comment, the model will happily continue to write code on the same line as the comment. Not an issue in a conversation model as there is a natural break in a conversation, but when integrating with tooling it's challenging.
Most likely the issue is that I'm not yet effective at prompt engineering, but I had no issues iterating on prompts for the earlier models. I'm loving the DaVinci model and it's working really well -- I just hope it's not discontinued too soon in favor of later models.