Using The Happy Hands Java Speech Editor

The Happy Hands Java Speech Editor written by Sean W Hennessy. He was a virtuosic piano player. He wrecked his hands because he was programming software at least 10 hours a day, then going home and banging on a piano for 2 hours a day. He wrote this program to try to salvage his career as a programmer, his piano, and hopefully make a significant contribution to the world.

Document conventions

Technical Overview

Requirements

• The recognizer properly matches what you are saying. Prefect code results.
• The recognizer decides it can't match with sufficient confidence. No action is taken.
• The recognizer incorrectly matches what you said. This usually results legitimate code, however not the code you intended. (Say "Undo Change" to undo this.)
• The recognizer properly matches what you said, and then imagines some trailing speech you didn't say. This results in the code you intended, plus some extra code you didn?t intend. Just remove this and continue.

At all the times Happy Hands in a set of speech handlers that are in a waiting state. A speech handler is an internal mechanism that may not seem important to the user, however since Happy Hands is a ground-breaking program in computer science, we feel that some technical explanation of these mechanisms is important. A speech handler is a collective set of behaviors and states that defines both the grammar to listen for and how to respond to the user's speech when it matches that grammar. It is the logical equivalent to the graphical dialog box that presents an input mechanism and handles input. In Happy Hands, a speech handler can be in an active or a waiting state. In the waiting mode, the handler is listening for speech that has a very simple grammar that's main purpose is to activate the handler. When it is activated, it changes its available grammars according to the context at hand. This two-stage mechanism is employed because it makes recognition easier on the underlying speech recognizer engine. Generally the activation grammars are simple and not context sensitive, however the grammars and the activated state may be very complex, containing thousands of alternative recognitions.

If you are considering trying Happy Hands and you have no recognizer installed at all, we sugest getting the Microsoft recognizer from their web site for free. This will enable you to try out Happy Hands without spending any money at all. On the other hand, Happy Hands works beautifully, so you might as well get Via Voice. Via Voice is generally a more conservative recognizer. It requires a greater confidence level to respond than the Microsoft recognizer. This means you get fewer false positives. The Microsoft recognizer is eager to respond, and is faster in deciding what you said, but sometimes responds to utterances it seems to imagine. Via Voice has the advantage of having a formal training system that makes you read it stories, so it will already know your voice when you start using Happy Hands. If you have already trained Via Voice for dictating letters, it is ready for Happy Hands.

Symbol renaming and remapping

Syntax Tree

Tree to text conversion

Text to tree conversion

File opening

File saving

Undo system

Projects

Testing

Tips and Tricks

Editing of existing elements

Grammar stringing

Reparsing and keyboard mode (the Esc key)

A transcriber is finished by starting another

Single utterance functions will not interfere

Always a waiting handler

Long names are better

Room noise

Default values

User Interface Components

Text editor component

There are four basic types of elements in a Happy Hands grammar. These elements are each rendered in a different way in the grammar view.

• The speech token element indicates an atomic word or phrase. This is rendered in the grammar view as a quoted string.
• The sequence element indicates a set of speech tokens to be said in right to left order. This is rendered in the grammar view as a horizontal row of elements.
• The alternatives element indicates a set of speech token alternatives. This is rendered in the grammar view as a vertical row of elements.
• The option element indicates that the enclosed element is optional in some way. This is rendered in the grammar view by enclosing the text of the enclosed element in parentheses or brackets. Parentheses mean zero or one elements maybe spoken. Braces mean zero or more elements may be spoken.

• The "Active" grammar view holds that active transcriber.
• The "Transcribers" holds the waiting transcribers.
• The "Functions" grammar view holds general speech activated functions.
• The "Types" grammar view holds the type sub grammar used within other grammars.
• The "Vars" grammar view holds the variables sub grammar used within other grammars.
• The "M.Fld" grammar view holds the member fields sub grammar used within other grammars.
• The "M.Meth" grammar view holds the member methods sub grammar used within other grammars.
• The "S.Fld" grammar view holds the selected fields sub grammar used within other grammars.
• The "S.Meth" grammar view holds the selected methods sub grammar used within other grammars.

Syntax tree view

Undo history view

Files view

Projects component

The project file setting controls the name of the file where project information is stored. When you create the new project, a file chooser appears to set this value. The value will also be added to the "settings/workspace.xml" file.

The source code directories setting is a list of directories where the source code of the project should be found. Each directory should go up to and including the first directory of the package. For example, if you have a package tree that goes pac_x.pac_y.pac_z, it is not necessary to make the directory end at the package root, pac_x. You could stop it at pac_y, and only these files will be included.

The "classpath" setting effects the following:

• It is given to the compiler when Happy Hands compiles your files and directories
• It is used to build up the speakable import grammars. If Happy Hands is unable to recognize the package names you are trying to import, make sure the package is included in the classpath setting.
• It is used to resolve type names and build grammars pertaining to the members of these types. This is very important, so get the classpath right.
• It is given to the Java VM when testing.

The "lib path" setting controls the dynamic link libraries or shared object libraries used by the Java virtual machine, when testing.

The "test class" setting is used when testing. It is the class you want the Java virtual machine to enter.

The "arguments" setting is used when testing. These are the arguments are given to Java program being tested.

Standard Grammars

Type grammar

Insert prefixes

• "Append" means the element will be inserted after the current element within the same block.
• "Insert" means the element will be inserted before the current element within the same block.
• "Insert in block" means the element will be inserted within the current block that is selected, or more precisely the current block owner, such as a for loop or method definition.
• "Enclose with" means that the current element will be enclosed with the insert block element.
• "Replace with" means that the current element will be replaced with the inserted element.

The Transcribers

Please note that in these descriptions we refer to specific utterances that they have since changed either because of implementation changes, or changes in user settings. The exact words that may be spoken at any point maybe discovered by looking at the grammar views.

Before this phrase is uttered, the package declaration transcriber is in a waiting state in the "Transcribers" grammar view. The grammar view shows that "append package declaration" is one of the grammars that the package listens for. Upon uttering the phrase, the package transcriber is moved to the active grammar view.

When the package declaration is inserted into the document, the first element of the package is selected. The package specifier defaults to "no_value". The package name is spoken in it's entirety. For example, to declare your file a member of the java.awt package, say "java a w t" in one utterance.

An incremental specification system may be used as well. Note that this feature has come and gone but may come back. When first element of the package specifier is selected, the transcriber is listening for any one of the root package specifiers found in the class path. Examples of root package specifiers are "java" and "javax". Upon recognition of one of these alternatives, another "no_value" package specifier is appended. The recognizer is now listening for package specifiers that are found in with in the first package specifier, for example "awt". Once you have spoken the last element of a package specifier, you need to manually delete the last element with the backspace key.

The red square around the class definition means the class definition transcriber is waiting for speech. The red square around the class name means that the naming transcriber is also waiting for speech. In Happy Hands, a transcriber may be assisted by a sub transcriber. In this case, the class definition transcriber is assisted by the naming transcriber.

Class name

It is also possible to give the name by speaking. The naming transcriber is started automatically when a new class is started. (The naming transcriber will start automatically only if the "auto name" option is enabled. It is disabled by default.) This transcriber accepts is a sequence of English words, or letters using the alpha-bravo-charlie alphabet. For example, to name your class "Sean", say "capital sierra echo alpha november". If the naming transcriber is not active, activate it by saying "class name", or click the mouse on the class name.

Access Modifier

Memory Modifier

Base Class

Interfaces

Inner elements

Example of how to create this class definition:

• Speak: "append class definition"
• Speak: "public extends type j button implements type action listener"
• Type: SubButton

Field declaration transcriber

Field Name

Access Modifier

Memory Modifier

Constant Modifier

Field Type

Example of how to create this field declaration:

• Speak: "append field declaration"
• Speak: "protected static type component" or "type component protected static" or "static type component protected"
• Type: defaultView

Method Name

Return Type

Access Modifier

Memory Modifier

Thread Modifier

Adding an argument

Adding a throws declaration

Base class override

Constructor

Example of how to create this method definition:

• Speak: "append method definition"
• Type: arraySet (anytime before adding arguments and name will stay selected)
• Speak: "public synchronized type integer" (pause anytime)
• Speak: "add argument type one dimensional integer array" (one utterance)
• Type: src
• Speak: "add argument type one dimensional integer array" (one utterance)
• Type: dst
• Speak: "add throws type array length exception" (one utterance)

Variable declaration transcriber

Expression statement transcriber

The expression transcriber is by far the most sophisticated transcriber in the Happy Hands system. It handles the transcription of all elements that may occur within an expression, including references to variable names, field names, method invocations, and object instantiations, binary operators, unary operators, tertiary operators, type casts, array initializations, anonymous classes, and more.

The expression transcriber is very context sensitive. You may notice a pause (of maybe a second) as the expression transcriber starts, because it is preparing the grammars speakable within the expressions context. The length of the pause will vary according to how much the context changed since the last time the expression transcriber was used.

The basic insert mode of the expression transcriber is replacement. Expressions are for the most part binary trees, with some lists too. When you utter an expression grammar, the currently selected expression element is usually replaced by what you have said. The exception to this rule is the utterance of operators, such as "addition", which adds siblings to the expression tree, and the grammars related to building method argument lists or array initialization lists. New expression parts requiring specification are called "no_value".

Example: the expression is formerly "[a]".
Say "addition" and it becomes "a + [no_value]".
To attach a binary operator to the left side of the currently selected value, say "left side ".

Example: the expression is formerly "[a]".
Say "left side addition" and it becomes "[no_value] + a".
To replace the currently selected value with a binary operator with default left and right values, say "replace with ".

Example: the expression is formerly "a + [b]".
Say "replace with addition" and it becomes "a + [no_value] + no_value".
Operator precedence is inferred from the selection state when operators are added.

Example: the expression is formerly "a + [b]".
Say "multiplication" and it becomes "a + b*[no_value].

Example: the expression is formerly "[a + b]".
Say "multiplication" and it becomes "(a + b)*[no_value].

Example: the expression is formerly "a * [b]".
Say "addition" and it becomes "a * (b+no_value)".

Example: the expression is formerly "[i]".
Say "post increment" and it becomes "[i++]".
Say "pre increment" and it becomes "[++i]".

Example: the expression is formerly "[a && b]".
Say "logical not" and it becomes "[!(a && b)]".

Field identifier for this object

Method invocation for this object

As with every other transcriber in Happy Hands, the method invocation transcriber pulls as much information as it can from the context of the code, rather than keeping track of the events that got it to its current state. This means that the overload switching grammars can be used even if the method was not just recently named.

After the method to invoke is identified, the first argument of the method is selected. (If the method has no arguments the method name element is selected.) A value can be uttered as in the standard expression transcription. To move to the next argument, say "next argument". To move to the previous argument, say "previous argument". You can also use the mouse or move the keyboard caret to do this.

Example: the expression is formerly "g.drawString(no_value, no_value, no_value)".
Say "variable text next argument variable x next argument variable y".
The expression becomes "g.drawString(text, x, y)".

Example: the expression is formerly "g.drawString(no_value, no_value, no_value)".
Say "variable text next argument, variable x addition integer two, next argument variable y multiplication integer negative one".
The expression becomes "g.drawString(text, x*2, y*-1)".

Within method argument lists, there is a special convenience that lets you move to the next argument automatically when the current argument is fully specified. If you say "sequence" at any time during an expression utterance, this puts the transcriber into a mode that automatically moves to the next argument when the current argument is fully specified. The special mode ends at the end of the utterance. Of course, it is not completely clear when an expression is fully specified. The transcriber uses the following rule: if the expression has no "no_value" elements in it, it is fully specified. Another value uttered after this, in sequence mode, will be applied to the next argument list element.

Example: the expression is formerly "g.drawString(no_value, no_value, no_value)".
Say "sequence variable text variable x variable y".
The expression becomes "g.drawString(text, x, y)".

Example: the expression is formerly "g.drawString(no_value, no_value, no_value)".
Say "sequence variable text variable x addition integer two variable y multiplication integer negative one".
The expression becomes "g.drawString(text, x, y)".
After the "addition", the second argument was "x + [no_value]". When "integer two" was processed, the current list element was not fully specified so the 2 was applied here rather than to the next argument.

Object instantiation

Example: "new type dimension sequence integer two integer three" gives "new Dimension(2, 3)".

String constant

You can use the normal expression grammar within a string by saying "append text" first. Example: the expression is formerly: "".
Say "append text variable apple"
The expression becomes "apple"
Create the expression "apple="+apple, type the =, then say "addition variable apple".

Anonymous class

Removing expression elements

Example: apple = orange*banana
Speak: "variable apple assignment variable orange multiplication field banana"

Example: apple = (orange+banana)*5
Speak: "variable apple assignment variable orange addition field banana" (pause anytime)
Select the addition element by moving the mouse over the +.
Speak: "multiplication integer 5". The () are added automatically.

Example: apple =orange+banana*5
Speak: "variable apple assignment variable orange addition field banana" (pause anytime)
Leave banana selected
Speak: "multiplication integer 5"

Example: kiwiFruit = new Fruit()
Speak: "kiwi fruit assignment new type fruit"

Example: kiwiFruit = new Fruit("kiwi", 10)
Speak: "kiwi fruit assignment new type fruit two arguments"
Speak: "string constant"
Type: kiwi
Speak: "integer 10"

Example: kiwiFruit.size = 20
Speak: "kiwi fruit chose member" (required pause)
Speak: "select field size assignment integer 20"

Example: kiwiFruit.size = 20
Speak: "kiwi fruit size assignment integer 20"

Example: fruitArray = new Knife().cut(kiwiFruit, 0.5f)
Speak: "fruit array assignment new type knife chose member" (required pause)
Speak: "select method cut" (good time to pause)
Speak: "sequence variable kiwi fruit floating point zero point five"

Example: fruitArray[0].size = banana
Speak: "variable fruit array index array integer zero"
Move selection to hold [fruitArray[0]]
Speak: "chose member" (require pause)
Speak: "select field size assignment variable banana"

            if([no_value]){
            }
	

The element filling in the test expression is selected. Speak an expression to fill in the value.

Example: if(a != b){...}
Speak: "append if test" (Required pause)
Speak: "variable a is not equal to variable b"

All of the transcribers that generate a block offer an additional insert prefix: "enclose with". Using this prefix will cause the selected element(s) to be enclosed with the block element.

Example: the expression is formerly: [System.out.println("debugging string")]
Speak: "enclose with if test" (Required pause) Speak: "variable b debug" The code becomes:

         if(bDebug){
              System.out.println("debugging string")
         }
	

Else if test transcriber

        else if([no_value]){
        }
	

Else block transcriber

        else{
        }
	

While loop transcriber

        while([no_value]){
        }
	

Do while loop transcriber

        do {
        } while(no_value);
	

For loop transcriber

        for(no_value; no_value; no_value){
        }
	

Switch block transcriber

        switch([no_value]){
        }
	

Case block transcriber

	
        case [no_value]:{
        }
	

Return transcriber

	
        return [no_value];
	

Throw transcriber

	
        throw [no_value];
	

Break transcriber

	
		break [no_target];
	

The continue transcriber

	
        continue [no_target];
	

The label transcriber

	

        [no_name]{
        }
	

The try catch block transcriber

	
        try{
        } 
        catch(no_type [no_name]){
        }
	

Example:

        try{
        } 
        catch(FileNotFoundException ex){
        }
	

Catch block transcriber

	
        catch(no_type [no_name]){
        }			
	

Finally block transcriber

	
        finally{
        }
	

Synchronized block transcriber

	
        synchronized([no_value]){
        }	
	

The doc comment transcriber

	
        /**
         * 
         */
	

The star comment transcriber

	
        /*
         * 
         */
	

The line comment transcriber

	
        //
	

Other speech activated functions

Open file

Save top file

Save file as

Close top file

Save and close top file

Save all files

Revert to auto save file

Revert file

Compile top file

Compile directory

Cancel Build

Format Directory

Start test

Stop test

Goto line

Show methods

Show fields

Show methods of type

Show fields of type

Use speech entry

Use keyboard entry

Reinitialize speech

Add to selection

Add to selection range

Clear selection

Code to comments

Comments to code

Delete selection

Copy selection

Paste copy

• "Append copy": appends the elements in the copy set after the currently selected element.
• "Insert copy": inserts the elements in copy set before the currently selected element
• "Insert in block copy": inserts the elements in the copy set within the currently selected block. If the currently selected element is not a block, the owning block of the currently selected element is found and the insert takes place within this.
• "Replace with copy": replaces the currently selected element with the elements in the copy set.

Edit element

Undo

Parse file

Parse element

Regenerate file

Page up

Page down

Exit Happy Hands

User Defined Macros

To define your own macro, you need to write a small Java class the uses the small macro API. You then put the fully qualified class name in the file settings/macros, and change the class path in the Happy Hands start up script to include your macro. The macro API is this:

	package lde;
	
	public class Macro{
		public static final String VALUE = "Value";

		/**
		 * Makes a macro
		 * @param name The macro name that is spoken
		 */		
		public Macro(String name);

		/**
		 * Add an argument to collect with a grammar appropriate to the type.
		 * Only type==VALUE is supported now.
		 */		
		public void addArgument(String name, String type);

		/**
		 * Add an argument to collect with a grammar appropriate to the type.
		 * Only type==VALUE is supported now.
		 */	
		public void setMacroName(String name);

		/**
		 * Retrieves the value of the argument of the given name
		 */
		public String getArgument(String name);

		/**
		 * Called by the macro manager to get the current macro text.  This should be
		 * overridden to build the text based on some constants and the argument values.
		 * This is called when the macro is inserted the first time and every time a value
	 	 * is specified by the user.
		 */
		public String makeText();
	}
	

Debug print out

Loop over length

		for(int i = 0; i < no_value.length; i++){
		}  
	

Loop over size

		for(int i = 0; i < no_value.size(); i++){
		}