Package edu.tufts.hrilab.vision

Class VisionComponent

java.lang.Object
edu.tufts.hrilab.diarc.DiarcComponent
edu.tufts.hrilab.vision.VisionComponent
All Implemented Interfaces:
VisionInterface, VisionInterface
public class VisionComponent extends DiarcComponent implements VisionInterface
DIARC Vision component.

  • Constructor Details

    • VisionComponent

      public VisionComponent()

  • Method Details

    • init

      protected void init()
      Description copied from class: DiarcComponent
      Perform any component initialization. This should not be used for setting local fields to default values as was the case in DIARC. This method is called after the constructor, and after parseArgs. Setting default values should be done in field declaration or in the constructor.
      Overrides:
      init in class DiarcComponent

    • reinitializeVisionComponent

      public void reinitializeVisionComponent(double camRotX, double camHeight)
      Should only be used for running component tests to reinitialize capture setting (e.g., use a different image, or camera mode).
      Parameters:
      camRotX -
      camHeight -

    • executionLoop

      protected void executionLoop()
      Description copied from class: DiarcComponent
      This is where any code that needs to be called repeatedly in a dedicated thread should live. It is not called by default and you must set shouldRunExecutionLoop to true in your constructor.
      Overrides:
      executionLoop in class DiarcComponent

    • pauseCapture

      public void pauseCapture()
      Specified by:
      pauseCapture in interface VisionInterface

    • resumeCapture

      public void resumeCapture()
      Specified by:
      resumeCapture in interface VisionInterface

    • stopAllSearches

      public void stopAllSearches()
      Specified by:
      stopAllSearches in interface VisionInterface

    • restartAllStoppedSearches

      public void restartAllStoppedSearches()
      Specified by:
      restartAllStoppedSearches in interface VisionInterface

    • shutdownComponent

      public void shutdownComponent()
      Description copied from class: DiarcComponent
      Any component that needs particular shutdown logic needs to override this method and
      Overrides:
      shutdownComponent in class DiarcComponent

    • additionalUsageInfo

      protected List<org.apache.commons.cli.Option> additionalUsageInfo()
      Description copied from class: DiarcComponent
      Command line options available in sub-class. This should be paired with a parseArgs implementation.
      Overrides:
      additionalUsageInfo in class DiarcComponent
      Returns:

    • parseArgs

      protected void parseArgs(org.apache.commons.cli.CommandLine cmdLine)
      Description copied from class: DiarcComponent
      Called directly after construction to pass runtime values that will override default values. This should parse all the options that additionalUsageInfo provides.zs
      Overrides:
      parseArgs in class DiarcComponent

    • startType

      public void startType(long typeId)
      Description copied from interface: VisionInterface
      Start detecting/tracking specified SearchManager.
      Specified by:
      startType in interface VisionInterface

    • stopType

      public void stopType(long typeId)
      Description copied from interface: VisionInterface
      Stop detecting/tracking specified SearchManager.
      Specified by:
      stopType in interface VisionInterface

    • stopAndRemoveType

      public void stopAndRemoveType(long typeId)
      Description copied from interface: VisionInterface
      Turns off search for typeId, and removes SearchManager.
      Specified by:
      stopAndRemoveType in interface VisionInterface

    • getAvailableTypeIds

      public List<Long> getAvailableTypeIds()
      Description copied from interface: VisionInterface
      Get List of all SearchManager IDs in long term memory.
      Specified by:
      getAvailableTypeIds in interface VisionInterface
      Returns:
      List of MemoryObjectType IDs

    • getTypeIds

      public List<Long> getTypeIds(double conf)
      Description copied from interface: VisionInterface
      Find all SearchManagers containing at least one token in STM. One each, regardless of how many tokens of a particular type are in STM.
      Specified by:
      getTypeIds in interface VisionInterface
      Parameters:
      conf - minimum confidence of tokens to be included
      Returns:
      SearchManager IDs

    • getTypeId

      public Long getTypeId(List<? extends Symbol> descriptors)
      Description copied from interface: VisionInterface
      Get the SearchManager ID (search ID) based on a Predicate description of the object. The Predicate description must match the SearchManager description exactly. If an existing SearchManager does not match the description, a new SearchManager will be attempted to be built. This assumes that all descriptors refer to the same object. Also starts the SearchManager if not running. NOTE: currently just ignores descriptors that vision doesn't know how to handle.
      Specified by:
      getTypeId in interface VisionInterface
      Parameters:
      descriptors - Predicate description of object
      Returns:
      SearchManager Id (-1 if there is no match)

    • getTypeId

      public Long getTypeId(Symbol objectRef)
      Description copied from interface: VisionInterface
      Get the SearchManager ID (search ID) based on a Predicate description of the object. The Predicate description must match the SearchManager description exactly. If a current SearchManager does not match the description a new SearchManager will be attempted to be built. Also starts the SearchManager if not running. NOTE: This is only a convenience method so a List doesn't need to be passed when only a single Predicate is used.
      Specified by:
      getTypeId in interface VisionInterface
      Parameters:
      objectRef - Predicate description of object
      Returns:
      SearchManager Id (-1 if there is no match)

    • nameDescriptors

      public boolean nameDescriptors(List<? extends Symbol> descriptors, Symbol typeName)
      Description copied from interface: VisionInterface
      Name the collection of Predicate descriptors. The descriptors must belong to a single SearchManager that already exists, or an attempt to create a new SearchManager matching the descriptors will be made. The description must match the SearchManager description exactly.
      Specified by:
      nameDescriptors in interface VisionInterface
      Parameters:
      descriptors - Predicate description of object
      typeName - Predicate name of object (name will be bound to typeId in VisionComponent)
      Returns:
      true if descriptors map to an existing typeId, false otherwise

    • nameDescriptors

      public boolean nameDescriptors(Long typeId, Symbol typeName)
      Description copied from interface: VisionInterface
      Name the collection of Predicate descriptors referred to by the SearchManager ID (search ID).
      Specified by:
      nameDescriptors in interface VisionInterface
      Parameters:
      typeId - SearchManager ID (search ID)
      typeName - Predicate name of object (name will be bound to typeId in VisionComponent)
      Returns:
      true if typeId exists, false otherwise

    • getDescriptors

      public List<Term> getDescriptors(Long typeId)
      Description copied from interface: VisionInterface
      Get the descriptors for a particular SearchManager.
      Specified by:
      getDescriptors in interface VisionInterface
      Returns:

    • getTokenIds

      public List<Long> getTokenIds(double conf)
      Description copied from interface: VisionInterface
      Get MemoryObject IDs in STM with a confidence level over the specified threshold. Only considers results from SearchManager searches that are currently running.
      Specified by:
      getTokenIds in interface VisionInterface
      Parameters:
      conf - minimum confidence of tokens to include
      Returns:
      List of STM MemoryObject IDs

    • getTokenIds

      public List<Long> getTokenIds(long typeId, double conf)
      Description copied from interface: VisionInterface
      Get MemoryObject IDs that have specified SearchManager ID with a confidence level above the specified threshold. If the specified SearchManager exists but is not running, it will be started before returning results.
      Specified by:
      getTokenIds in interface VisionInterface
      Parameters:
      typeId - SearchManager ID of tracked objects
      conf - minimum confidence of tokens to include
      Returns:
      List of STM MemoryObject IDs

    • getTokenIds

      public List<Long> getTokenIds(List<? extends Symbol> descriptors, double conf)
      Description copied from interface: VisionInterface
      Get MemoryObject IDs for all tokens *exactly* matching descriptors in STM with confidence greater than threshold. If a matching type is found, it will be started (if not already running). If no match exists, a SearchManager will attempt to be built and started before returning. This is largely a convenience method and getTypeId should be used whenever possible.
      Specified by:
      getTokenIds in interface VisionInterface
      Parameters:
      descriptors - list of Symbol descriptors
      conf - the minimum confidence level
      Returns:
      List of STM MemoryObject IDs

    • getTokens

      public List<MemoryObject> getTokens(double conf)
      Description copied from interface: VisionInterface
      Get MemoryObjects in STM with a confidence over the specified threshold. Only considers results from SearchManager searches that are currently running.
      Specified by:
      getTokens in interface VisionInterface
      Parameters:
      conf - minimum confidence of tokens to include
      Returns:
      List of MemoryObjects

    • getTokens

      public List<MemoryObject> getTokens(long typeId, double conf)
      Description copied from interface: VisionInterface
      Get MemoryObjects of the specified SearchManager ID with confidence above specified threshold. If the specified SearchManager exists but is not running, it will be started before returning results.
      Specified by:
      getTokens in interface VisionInterface
      Parameters:
      typeId - SearchManager ID of tracked objects
      conf - minimum confidence of tokens to include
      Returns:
      List of MemoryObjects

    • getTokens

      public List<MemoryObject> getTokens(List<? extends Symbol> descriptors, double conf)
      Description copied from interface: VisionInterface
      Get MemoryObjects *exactly* matching descriptors in STM with confidence greater than threshold. If a matching type is found, it will be started (if not already running). If no match exists, a SearchManager will attempt to be built and started before returning. This is largely a convenience method and getTypeId should be used whenever possible.
      Specified by:
      getTokens in interface VisionInterface
      Parameters:
      descriptors - list of Symbol descriptors
      conf - the minimum confidence level
      Returns:
      List of STM MemoryObjects

    • getToken

      public MemoryObject getToken(long tokenId, double conf)
      Description copied from interface: VisionInterface
      Get the MemoryObject with the specified id.
      Specified by:
      getToken in interface VisionInterface
      Parameters:
      tokenId - MemoryObject ID in STM
      conf - minimum confidence of tokens to include
      Returns:
      MemoryObject token (Null if doesn't exist)

    • createNewType

      public Long createNewType()
      Description copied from interface: VisionInterface
      This instantiates a new MemoryObjecType (search manager) which can be incrementally configured via addDescriptor.
      Specified by:
      createNewType in interface VisionInterface
      Returns:
      SearchManager ID (search id)

    • addDescriptor

      public boolean addDescriptor(long typeId, Symbol descriptor)
      Description copied from interface: VisionInterface
      Add new search constraint (i.e., ImageProcessor) to an existing SearchManager (specified by searchID).
      Specified by:
      addDescriptor in interface VisionInterface
      Parameters:
      typeId - - unique ID returned by createNewType
      descriptor - - predicate describing visual search attribute (e.g., "red", "round")

    • removeDescriptor

      public boolean removeDescriptor(long typeId, Symbol descriptor)
      Description copied from interface: VisionInterface
      Remove search constraint (i.e., processing descriptor and/or ImageProcessor) to an existing SearchManager (specified by searchID).
      Specified by:
      removeDescriptor in interface VisionInterface
      descriptor - - to remove
      Returns:
      - if removal was successful

    • endDescriptorChanges

      public void endDescriptorChanges(long typeId)
      Description copied from interface: VisionInterface
      Signal the end of constraint addition. Descriptors/constraints can no longer be added to the SearchManager after this has been called. This also starts the Detector and Tracker and will choose the "object" Detector if one hasn't already been selected.
      Specified by:
      endDescriptorChanges in interface VisionInterface

    • confirmToken

      public boolean confirmToken(long tokenId)
      Description copied from interface: VisionInterface
      Confirms that the object is still in STM.
      Specified by:
      confirmToken in interface VisionInterface
      Parameters:
      tokenId - MemoryObject ID of the object to be confirmed
      Returns:
      true if the MemoryObject is present in STM

    • confirmToken

      public boolean confirmToken(MemoryObject token)
      Description copied from interface: VisionInterface
      Confirms that the object is still in STM.
      Specified by:
      confirmToken in interface VisionInterface
      Parameters:
      token - MemoryObject to be confirmed
      Returns:
      true if the object is present in STM

    • takeSnapshot

      public void takeSnapshot(String filename)
      Description copied from interface: VisionInterface
      Save the current frame to file (e.g., for logging).
      Specified by:
      takeSnapshot in interface VisionInterface

    • getFrame

      public byte[] getFrame()
      Description copied from interface: VisionInterface
      Return the camera's last frame in a byte array.
      Specified by:
      getFrame in interface VisionInterface
      Returns:
      the image in a byte array

    • getDisparityFrame

      public byte[] getDisparityFrame()
      Description copied from interface: VisionInterface
      Return the camera's last disparity frame in a byte array.
      Specified by:
      getDisparityFrame in interface VisionInterface
      Returns:

    • getDepthFrame

      public byte[] getDepthFrame()
      Description copied from interface: VisionInterface
      Return the camera's last depth frame in a byte array.
      Specified by:
      getDepthFrame in interface VisionInterface
      Returns:

    • getImageSize

      public Dimension getImageSize()
      Description copied from interface: VisionInterface
      Returns the size of the image being used.
      Specified by:
      getImageSize in interface VisionInterface
      Returns:
      Dimension containing the size

    • getUnsatisfiableConstraints

      public List<Symbol> getUnsatisfiableConstraints(List<? extends Symbol> descriptors)
      Description copied from interface: VisionInterface
      Find what part(s) of the requested descriptors vision doesn't know how to process.
      Specified by:
      getUnsatisfiableConstraints in interface VisionInterface
      Returns:
      list of (possibly empty) unsatisfiable constraints

    • getUnsatisfiableConstraints

      public List<Symbol> getUnsatisfiableConstraints(Symbol descriptor)
      Description copied from interface: VisionInterface
      Find what part(s) of the requested descriptor vision doesn't know how to process.
      Specified by:
      getUnsatisfiableConstraints in interface VisionInterface
      Returns:
      list of (possibly empty) unsatisfiable constraints

    • learn

      public boolean learn(Term learningTerm, List<Map<Variable,Symbol>> bindings)
      Description copied from interface: VisionInterface
      Use the passed in term to learn a new descriptor. The term must have two arguments, the first arg 'describing' the new thing-to-learn, and the second arg 'naming' the new thing-to-learn. The description must be composed entirely of things vision already knows about.
      Specified by:
      learn in interface VisionInterface
      Parameters:
      learningTerm - term specifying the kind of learning to do (e.g., instanceOf(X,Y), definitionOf(X,Y))
      bindings - variable bindings to any variables in query term
      Returns:
      if thing-to-learn was successfully learned

    • unlearn

      public boolean unlearn(Term learnTerm, List<Map<Variable,Symbol>> bindings)
      Description copied from interface: VisionInterface
      Un-learns things that have been learned by the learn method.
      Specified by:
      unlearn in interface VisionInterface
      Returns:

    • makeObservations

      public List<Map<Variable,Symbol>> makeObservations(Term observation)
      Description copied from interface: VisionInterface
      Entry point in vision for making world observations.
      Specified by:
      makeObservations in interface VisionInterface
      Parameters:
      observation - single term specifying observation to make
      Returns:
      bindings for any free variables in observation term

    • writeReferencesToFile

      public void writeReferencesToFile(String filename)
      Write vision references to file.
      Parameters:
      filename -

    • loadReferencesFromFile

      public void loadReferencesFromFile(String filename)
      Load pre-defined vision references from file.
      Parameters:
      filename -

    • sandbox

      public void sandbox()

    • checkObjAt

      public List<Map<Variable,Symbol>> checkObjAt(Predicate predicate)