Package edu.tufts.hrilab.llm.serviceproviders.tr.social

Class SocialTRLLMServiceProvider

java.lang.Object
edu.tufts.hrilab.diarc.DiarcComponent
edu.tufts.hrilab.llm.LLMServiceProvider
edu.tufts.hrilab.llm.serviceproviders.tr.social.SocialTRLLMServiceProvider
Direct Known Subclasses:
SocialAvaLLMServiceProvider, SocialSpotLLMServiceProvider
public abstract class SocialTRLLMServiceProvider extends LLMServiceProvider

  • Field Details

    • currentChatCollections

      protected final Map<org.apache.commons.lang3.tuple.Pair<Symbol,Symbol>,ChatCollection> currentChatCollections
      Maintained set of ChatCollections, split by user/AgentTeam pair Contains all relevant conversation contexts and their managed histories between each pair

    • chatContextHistories

      protected final Map<org.apache.commons.lang3.tuple.Pair<Symbol,Symbol>,List<ChatContext>> chatContextHistories
      Extended History of chat contexts, stored for reference from ITK actions for the LLM to conditionally retrieve information about past interactions without cluttering current chat context.

    • promptVars

      protected Map<String,String> promptVars
      Prompt variables to be replaced in sent Completion requests. Refer to the Prompt class and apply method if this notion is unfamiliar to you.

  • Constructor Details

    • SocialTRLLMServiceProvider

      public SocialTRLLMServiceProvider()

  • Method Details

    • initDomainPromptVars

      protected abstract Map<String,String> initDomainPromptVars()

    • updateDomainPromptVars

      protected abstract Map<String,String> updateDomainPromptVars(Utterance utterance, boolean parsing)

    • getRefTypes

      protected abstract List<String> getRefTypes()
      List of types handled by consultants in the system - when encountering symbols with these types during NLG preprocessing/NLU postprocessing, replaces references with descriptors/typed descriptors with variables.

    • getNluSystemPrompt

      protected Prompt getNluSystemPrompt()

    • getNlgSystemPrompt

      protected Prompt getNlgSystemPrompt()

    • getValidationNLUSystemPrompt

      protected Prompt getValidationNLUSystemPrompt()

    • getValidationNLGSystemPrompt

      protected Prompt getValidationNLGSystemPrompt()

    • getQuestionAnswerSystemPrompt

      protected Prompt getQuestionAnswerSystemPrompt()

    • getGoalTranslationPrompt

      protected Prompt getGoalTranslationPrompt()

    • generateInstructTools

      protected abstract List<Tool> generateInstructTools(boolean xml)

    • generateITKTools

      protected abstract List<Tool> generateITKTools()

    • generateReplyTools

      protected abstract Map<String,Tool> generateReplyTools()

    • generateFocusedContextChatTemplates

      protected abstract Map<String,ChatTemplates> generateFocusedContextChatTemplates()

    • correctCommonNLUMistakes

      protected abstract String correctCommonNLUMistakes(Utterance utterance)

    • init

      public void init()
      Description copied from class: DiarcComponent
      Perform any component initialization. This should not be used for setting local fields to default values as was the case in DIARC. This method is called after the constructor, and after parseArgs. Setting default values should be done in field declaration or in the constructor.
      Overrides:
      init in class LLMServiceProvider

    • executionLoop

      protected void executionLoop()
      Description copied from class: DiarcComponent
      This is where any code that needs to be called repeatedly in a dedicated thread should live. It is not called by default and you must set shouldRunExecutionLoop to true in your constructor.
      Overrides:
      executionLoop in class LLMServiceProvider

    • initPromptVars

      protected Map<String,String> initPromptVars()

    • updatePromptVars

      protected void updatePromptVars(Utterance utterance, boolean parsing)
      Update prompt variables whose values need to be explicitly retrieved at call time TODO: a lot of these can be updated to be held internally and updated on events

    • updateFocusedContextPromptVars

      protected Map<String,String> updateFocusedContextPromptVars(Symbol user, Symbol agent)

    • getContextDependentTools

      protected Map<String,Tool> getContextDependentTools(Symbol agentTeam)

    • parseUtteranceLLMService

      public Utterance parseUtteranceLLMService(Utterance utterance)
      Specified by:
      parseUtteranceLLMService in class LLMServiceProvider

    • parseUtteranceLLMService

      public Utterance parseUtteranceLLMService(Utterance utterance, boolean useStreaming)
      Specified by:
      parseUtteranceLLMService in class LLMServiceProvider

    • parseUtteranceLLMService

      public Utterance parseUtteranceLLMService(Utterance utterance, boolean useStreaming, boolean reattemptedParse)
      Main NLU method used by LLMParserComponent
      Parameters:
      utterance - Utterance with populated words to parse to semantics
      reattemptedParse - Whether this is a followup call after negative user validation (so we do not update the Chat history with the same utterance more than once)
      Returns:
      The utterance with populated semantics, or null if no valid parse is found

    • handleToolResponse

      public Utterance handleToolResponse(Completion llmResponse, Utterance inputUtterance, ChatCollection chatCollection, List<ToolDefinition> advertisedToolList, StringBuilder errorResponseBuilder, boolean useStreaming)
      Method for handling Completion responses containing Tool calls, 3 main cases: - Base: Return an Utterance populated with the Tool's underlying semantics. In the case of an external cache hit, the context will be discarded and handleValidatedToolResponse will be called from the corresponding dialogue history update. - LLM ITK - will return semantics for goal without validation (Underlying semantics should match a SKIPPENDING priority goal which calls addToolResponse(Symbol, Symbol, String, String)) - Invalid parse - Add a system error message to the Chat, call the completion again, and return a text response to the user with the result - *NEW* Stand-in tool(s) for text responses - Experimenting targeting a specific tool to give text responses rather than using handleTextResponse above with promising initial success. Allowing the LLM to target text responses or tool calls was giving it a hard time calling tools rather than just saying it performed the underlying functionality.

    • handleTextResponse

      public Utterance handleTextResponse(Completion llmResponse, Utterance inputUtterance, ChatContext chatContext, StringBuilder errorResponseBuilder, boolean streaming)
      Method for handling Completion responses containing Text responses. Add directly as assistant message to history and return generateResponseFromString semantics with needsValidation set to false

    • handleValidationResponse

      public Utterance handleValidationResponse(Completion llmResponse, Utterance inputUtterance, ChatCollection chatCollection, List<ToolDefinition> advertisedToolList, StringBuilder errorResponseBuilder, boolean useStreaming)
      Method for handling Completion responses containing during validation.

    • inExtendedParseLoop

      public boolean inExtendedParseLoop(Symbol user, Symbol agentTeam)
      Method for HybridParser/NLG components to prevent use of other parser/NLG components once entering an extended interaction
      Specified by:
      inExtendedParseLoop in class LLMServiceProvider

    • usedLLMParse

      public void usedLLMParse(Symbol user, Symbol agentTeam)

    • getLLMConversationLength

      public int getLLMConversationLength(Symbol user, Symbol agentTeam)
      Method used in nlu.asl to understand on positive validation of a command whether the command was a first-time instruction or part of an extended, contextual dialogue. Necessary to know what Utterance's are safe to cache.
      Overrides:
      getLLMConversationLength in class LLMServiceProvider

    • reduceChatContextHistory

      public void reduceChatContextHistory()

    • reattemptParse

      public boolean reattemptParse(Utterance utterance, boolean streamingTTS)
      Method called from nlu.asl after negative validation of an LLM parse. Adds a system message informing the LLM of what occurred and calling the completion endpoint again with the user response. May result in an updated tool call or a text response for the user.

    • convertSemanticsToTextLLMService

      public Utterance convertSemanticsToTextLLMService(Utterance utterance)
      Specified by:
      convertSemanticsToTextLLMService in class LLMServiceProvider

    • convertSemanticsToTextLLMService

      public Utterance convertSemanticsToTextLLMService(Utterance utterance, boolean streamingSayText)
      Main NLG method used by LLMNLGComponent
      Specified by:
      convertSemanticsToTextLLMService in class LLMServiceProvider
      Parameters:
      utterance - Utterance with populated semantics to transform to words
      streamingSayText - - whether the response should be feed incrementally to TTS using a streaming Completion
      Returns:
      The utterance with populated words, or null if no generation could be formed

    • focusDialogueChat

      public Utterance focusDialogueChat(Symbol user, Symbol agentTeam, String contextIdentifier, long cid)
      Transfer to a focused context chat. Only called through runContextSpecificChat in nlu.asl - specific assumptions and logic made around this method, do not call this method from elsewhere without fully understanding what you are doing.

    • interruptWaitOnFocusedChat

      public boolean interruptWaitOnFocusedChat(Symbol user, Symbol agentTeam, long cid)
      Interruption method for waitOnFocusedChat(Symbol, Symbol, long) Unlocks current chatContext for user/agentTeam pair if matching supplied Context ID

    • waitOnFocusedChat

      public void waitOnFocusedChat(Symbol user, Symbol agentTeam, long cid)
      Method to wait for termination of a focused ChatContext; currently only called from runContextSpecificChat in nlu.asl.
      Parameters:
      user - external interactor who is a part of this conversation
      agentTeam - DIARC AgentTeam which is a part of this conversation
      cid - unique identifier for this focused chat context ("context ID" as in DIARC Context class ID)

    • getContextTransferResponseUtterance

      public Utterance getContextTransferResponseUtterance(long cid)
      Utility to get second LLM response on transfer to a focused context (see comments above contextTransferResponseUtterance for an explanation)

    • callChatCompletion

      protected Completion callChatCompletion(Chat chat, List<ToolDefinition> tools)

    • streamingChunkHandler

      public void streamingChunkHandler(OpenaiChatCompletionResponse lastChunk, OpenaiChatCompletionResponse currentResponse, Chat currentChat)

    • callChatCompletionStreaming

      protected Completion callChatCompletionStreaming(Chat chat, List<ToolDefinition> tools, long uid)

    • endChatContext

      public void endChatContext(Symbol user, Symbol agentTeam)
      Method to discard the current chat context. Currently only used when validation questions expire (should be used for general nonactivity timeout as well)

    • getChatCollection

      public ChatCollection getChatCollection(Symbol user, Symbol agentTeam)

    • callChatCompletionForTextResponse

      public String callChatCompletionForTextResponse(Symbol user, Symbol agentTeam, long id)

    • addVisionResponse

      public Utterance addVisionResponse(Symbol user, Symbol agentTeam, String visionData, String toolCallName)

    • addToolResponse

      public Utterance addToolResponse(Symbol user, Symbol agentTeam, String toolResponse, String toolCallName)
      Method used by "social LLM" itk behaviors (llm to/from diarc itk) to add DIARC's response to an ITK request and then call the Completion endpoint with the updated Chat for a text response.

    • suspendLLMChatContext

      public Justification suspendLLMChatContext(Symbol user, Symbol agentTeam, long cid)

    • resumeLLMChatContext

      public Justification resumeLLMChatContext(Symbol user, Symbol agentTeam, long cid)

    • cancelLLMChatContext

      public Justification cancelLLMChatContext(Symbol user, Symbol agentTeam, long cid)

    • getCurrentGoals

      protected List<Predicate> getCurrentGoals(Symbol agent)

    • getPendingGoals

      protected List<Predicate> getPendingGoals(Symbol agent)

    • getScheduledGoals

      protected List<Predicate> getScheduledGoals(Symbol agent)

    • getCurrentGoalsSummary

      protected String getCurrentGoalsSummary(Symbol agent)

    • getFullGoalsSummary

      public String getFullGoalsSummary(Symbol agent)

    • getExtendedConversationHistories

      public List<ChatContext> getExtendedConversationHistories(Symbol user, Symbol agentTeam)

    • getExtendedConversationHistoryString

      public String getExtendedConversationHistoryString(Symbol user, Symbol agentTeam)

    • getCurrentQuestionsString

      protected String getCurrentQuestionsString(Symbol agent)

    • getLearnedActionPredicates

      public List<Predicate> getLearnedActionPredicates()

    • initLearnedActions

      protected void initLearnedActions()
      Overrides:
      initLearnedActions in class LLMServiceProvider

    • handleDatabaseAddition

      public void handleDatabaseAddition(ActionDBEntry entry) throws ai.thinkingrobots.trade.TRADEException
      Overrides:
      handleDatabaseAddition in class LLMServiceProvider
      Throws:
      ai.thinkingrobots.trade.TRADEException

    • handleDatabaseRemoval

      public void handleDatabaseRemoval(ActionDBEntry entry) throws ai.thinkingrobots.trade.TRADEException
      Overrides:
      handleDatabaseRemoval in class LLMServiceProvider
      Throws:
      ai.thinkingrobots.trade.TRADEException

    • getInstructableActions

      public List<String> getInstructableActions()

    • updateLLMConversationHistory

      public void updateLLMConversationHistory(Utterance utterance, boolean newUtterance)
      Experimenting adding messages to ChatContexts from dialogue history updates (**currently NLG only) in case output from external parsers or generators is chosen over this one. (If intending to do the same for NLU, need to first stop adding messages directly in parseUtterance)
      Overrides:
      updateLLMConversationHistory in class LLMServiceProvider

    • questionAskingCallback

      public void questionAskingCallback(Symbol agent, boolean active, Question question)
      Question asking notification callback - throw away the current chat context if the associated system validation question timed out without a response.

    • processArbitraryToolArgument

      protected Symbol processArbitraryToolArgument(String argumentName, Object argumentValue, ToolParameterV2 argumentDefinition, String parentArgumentName, Symbol actor, Utterance.Builder builder, Map<Symbol,Variable> varMap, StringBuilder errorMessage, boolean unpopulatedArgSafe, boolean emptyArgAllowed)

    • processMapArgument

      protected Symbol processMapArgument(String name, Object arguments, ToolParameterV2 parameterDefinition, boolean includeActor, Symbol actor, Utterance.Builder builder, Map<Symbol,Variable> varMap, StringBuilder errorMessage, boolean unpopulatedArgSafe, boolean emptyArgAllowed)

    • processSymbolArgument

      protected Symbol processSymbolArgument(String argName, String argString, List<String> possible_values, StringBuilder errorMessage, Map<Symbol,Variable> varMap, Utterance.Builder builder, boolean inner, boolean unpopulatedArgSafe, boolean emptyArgAllowed)

    • getChatHistory

      public List<Message> getChatHistory(Symbol user, Symbol agentTeam)

    • getChatHistoryAsJSON

      public String getChatHistoryAsJSON(Symbol user, Symbol agentTeam)

    • getChatHistoryAsString

      public String getChatHistoryAsString(Symbol user, Symbol agentTeam)

    • getChatHistoryAsString

      public String getChatHistoryAsString(Chat chat)

    • getChatHistoryAsString

      public String getChatHistoryAsString(List<Message> messages, int numMessages)

    • getLastResponseString

      public String getLastResponseString(Symbol user, Symbol agentTeam)

    • getLastResponseStreamed

      public boolean getLastResponseStreamed(Symbol user, Symbol agentTeam)

    • toolToDescString

      protected String toolToDescString(Tool tool, boolean includePossibleValues)
      Convert a Tool definition to a description string for the LLM to use as reference - want to give the information of what may have been called by the system previously or what can be used as an inner argument to another tool without actually advertising these actions as valid Tools themselves (e.g. NLG, goal scheduling, action ITK) (TODO: keep an eye on this, this information can not be formatted in a way familiar to the LLM as a tool call, or else it will try to call it)

    • getContextTransferFillerPhrase

      public String getContextTransferFillerPhrase(String contextId, long cid, Symbol user, Symbol agent)

    • getBaseRoleDescription

      protected String getBaseRoleDescription()

    • getBaseContextualInformation

      protected String getBaseContextualInformation()

    • getBaseResponseConstraints

      protected String getBaseResponseConstraints()

    • getBaseResponseStyle

      protected String getBaseResponseStyle()

    • getFocusedResponseConstraints

      protected String getFocusedResponseConstraints()

    • getFocusedResponseStyle

      protected String getFocusedResponseStyle()

    • getFocusedResponsePolicy

      protected String getFocusedResponsePolicy()

    • getFocusedFinalInstruction

      protected String getFocusedFinalInstruction()