ABCI Workshop 2013:
Language Model and
Architecture for RSVP-iconCHAT
Karl Wiegand
Northeastern University
Boston, MA USA
January 14, 2013

My Background
Computer science Ph.D. student
Natural language processing (NLP)
Artificial intelligence (AI)
Applications to augmentative and alternative communication (AAC)

Outline
Constraints and approach
Interface and demo
Language model
Current architecture
Application to Unlock

Constraints and Approach
Constraints:
Single input signal (P300)
Icon-based AAC
Approach:
Event timer
Semantic frames

Semantic Frames
Actions are central to messages (Fillmore, 1976)
Verbs have "frames" with semantic roles:
Give ( Agent, Object, Beneficiary )
WordNet, FrameNet, "Read the Web"
Verb-first message construction (Patel et al, 2004)
Any order in RSVP-iconCHAT

Interface

Demonstration

Language Model
predict(role, state): listof([word, probability])
Semantic grams (Wiegand and Patel, 2012)
"I like to play chess with my brother."
brother, chess
brother, i
brother, like
brother, play
chess, i
...
brother, chess, i
brother, chess, like
brother, chess, play
chess, i, like
chess, i, play
...
Mention performance on missing word task, especially to n-gram approaches.

LM Training
Choose a corpus:"Blog Authorship Corpus""Crowdsourced AAC-Like Corpus"
Split sentences and remove stop words
Count sentence lengths
Stem and count sem-grams

LM Algorithm
Tag closed vocabulary with possible roles
Select statistics for closed vocabulary
Get words from target role
Generate sem-grams from current roles
Convert sem-gram counts into probabilities
Reorder and return

More LM
Semantic frames have syntactic forms
First-word prediction is based on 1-grams
Roles have uniform selection probability
How do we detect wrong selections of a role?  Of a word?

Current Architecture

Architecture Details
BCI/Host Controller ("The Brain")
Control loop and signal processing

Presentation Layer (Client)
User interaction -- images and keyboard

Language Model (Client)
Oval and word prediction
Semantic selections to syntactic phrase

Runtime Process: LM
Cache vocabulary statistics
Connect to the host controller
Wait for a request header:

		"Oval probabilities" -- None, current_state
"Icon probabilities" -- oval, current_state
"Syntactic utterance" -- current_state

Runtime Process: Presentation
Connect to host controller
Wait for a request header:

		"Start event loop" -- [oval/icon, bitcode]
		"Pause event loop"
		"Stop event loop"
		"Made decision" -- [oval/icon, bitcode]
		"Reset event loop" -- [oval/icon, bitcode]

Runtime Process: Host
Initialize gTec hardware
Initialize BCI modules
Receive connections from Client modules
Do:
Query LM for oval probabilities
Reorganize display order of ovals
Send display order to Presentation
Detect P300
Query LM for icon probabilities
Reorganize display order of icons
Send decision to Presentation
Repeat 4a - 4g until user selects Speak...

Runtime Process: Host (cont.)
Query LM for syntactic utterance
Send utterance to Presentation
Reset Presentation
Go to 4a...

Project Management
Git repository on BitBucket
Task management via Asana
Schedules in Google Calendar
Meeting notes in Google Drive
Code backups, relevant papers, and meeting board photographs in CSLftp

Implementation Details
IPC is via TCP/IP packets
Shared network packet structure
Controller uses Matlab
Presentation and LM use Python, Twisted, and either Pygame+SDL or Pyglet+OpenGL
Test Controller uses Python+Kivy+Twisted

Application to Unlock
P300 design is different than SSVEP design
Semantic frames:
Divide sentences
Free order construction
Semantic to syntactic mapping
Semantic grams:
Free order prediction
Require applicable corpus
IPC is nice with a standard packet structure

Thanks to Dr. Rupal Patel, Dr. Deniz Erdogmus, and the National Science Foundation (Grant #0914808).
Thank you for listening!