Suggesting sub-codes is not working

I am trying to use the AI-Assist function "Suggest subcodes" but my conclusion at this point is that it is unusable. Firstly because it produces inconsistent results that have nothing to do with my instructions and secondly because it does not reveal the system prompt that MaxQDA uses when sending the coded segments to the AI. I also suspect that that not all segments are sent to the AI (otherwise the absence of certain sub-code suggestions would be even more difficult to understand). 

Is anybody making similarly frostrating experiences with this feature? Has anybody figured out a way of getting more meaningful results? 

(As context: I am familiar with various prompting strategies and with how LLMs work, so please don't bother lecturing me on temperature settings or how I should try different prompts. I have already done that and the nonsensical results cannot be explained by my prompt alone but are more likely due to a combination of a bad system prompt (which MaxQDA doesn't reveal), bad choice of LLM, and incomplete submission of the text segements.

Just to give an example of the kind of nonsense it produces: I have a large number of text segments coded as "small talk" and I asked for sub-codes with the following instruction: "Find subcodes for all forms of communication that are distinguished from small talk". 

One of the responses started with "Here are some key points about the relationship between small talk and gossip based on the passage:"

While there is no reason why it should focus on the relationship between "gossip" and "small talk" given that the text snippets mention an abundance of other forms of talk. Furthermore, the wording "based on the passage" suggest that the submitted segments were not submitted as individual units of text but as a single chunk of text. If this is the case, this would be a serious design flaw. When the user submits a list of text segments and wants sub-codes based on these, this intent has to be conveyed to the AI either via the system prompt and the format of the submitted segments or by subitting one segment at a time.

When I submitted the same request with the same instructions again, it came back with "Here are the key points from the study on small talk in the workplace: ..." So, again, it is ignoring the instruction and treating the segments as a single piece of text (this time, it is a "study"). 

Other tries yielded somewhat more reasonable results (though still ignoring large parts text segments at hand). I noticed that the name of the main code seems to play a very strong role in the system prompt and it seems to insist that the sub-codes are actual sub-categories of the the main category. This made me wonder whether the AI might be confused by the default formar of autocodes (e.g. "Autocode - ANY: small talk"), so I removed "Autocode - ANY:" form the code name and tried again. - It did seem to make a bit of a difference, but it is hard to tell given that the results even with identical code name are so inconsistent.

I think it is crucial that the process of what gets submitted (including the system prompt) and which LLM model is used, becomes more transparent. Currently, the information available is not enough to assess why the "Suggest subcodes" feature is not working. Is it the system prompt, the LLM model, or the (selective) transmission of text segments? Or a combination of these? Will MaxQDA ever let users know?

0 Votes