ࡱ> %'$Y7 *"bjbjUU z67|7|pl &0 < $` P` ]8rDDDD|0000S0|b4|7$9 ;8 "8T|K8^T( |00/ ^ 0|8 8i` XGv/0|-80]8/<B<0` ` Effective Notification Systems Depend on User Trust Scott LeeTiernan, Edward Cutrell, Mary Czerwinski, and Hunter Hoffman Microsoft Research, One Microsoft Way, Redmond, WA 98052 USA Abstract: Intelligent messaging systems attempt to determine what information is important to a users task and when and how to interrupt the user with that information. Such systems are probabilistic, and will be wrong some percentage of the time. We conducted an experiment to assess the impact of variable reliability in a notification system on users trust and use of the system. We show how an unreliable system that violates users trust may lead to the abandonment of the system. This disuse behavior pattern persisted despite subsequent improvements in the reliability of the underlying intelligent system. Our results provide guidance in the design of notification user interfaces. Keywords: Instant messaging, empirical studies, interruptions, intelligent systems 1 Introduction The amount of information available for presentation to computer users is enormous. New email arrives constantly, help systems offer their services, and appointment and task reminders ensure we are at least aware that we are running behind. Intelligent systems (e.g., Horvitz, Jacobs and Hovel, 1999) have been developed to help manage the onslaught of potential incoming information. These systems filter messages, making decisions about what information is important, the optimal time for a notification, and how to display the message. Such systems face several difficult design problems associated with the user interface model, and there has been surprisingly little research to guide system designers. One psychological phenomenon central to designing a good user interface for notifications is the trust the user develops in the system. When an intelligent system makes mistakes, which are especially likely in the early going as it learns, users may place less trust in the messaging system. In the extreme case, users may adopt a strategy of completely ignoring or disabling the system. Maltz, and Meyer (2000) studied a demanding visual task in which potentially beneficial cues were provided. The cues varied in their validity from invalid, moderately valid, to highly valid cues, or there were no cues (control condition). By the second block of trials, only the participants receiving highly valid cues continued to utilize the cues. The question of users trust in the system therefore is important when designing a notification interface for a system known to be somewhat unreliable. If a first impression dominates subsequent interpretations, then the interface should strive to mitigate any negative first impressions. In this paper, we present a study of behavioral reactions to a system with changing reliability. What happens when a system is initially unreliable, but becomes more reliable later on? Once users trust of the notification interface has been broken will they ever reassess system reliability and update their behavior to incorporate new information? 2 Empirical Study Procedure Sixteen participants, ranging in age from 19 to 51, each completed 84 word puzzles similar to the game Boggle. Shown a 6 X 6 grid of letters, participants were given a specified time frame to find the 5-letter solution word beginning with the letter in bold. Periodically, the system sent notifications to participants that, if responded to, revealed the first three letters of the solution word. Notifications were either subtle or salient. Salient notifications consisted of a large spinning graphic shown near screen-center, accompanied by a loud sound. Subtle notifications were smaller graphics shown at the lower right of the screen, accompanied by a quiet sound. Participants were told that a salient notification meant the computer thought the message was helpful to the current task, while a subtle notification meant the system thought the incoming message was not relevant to the current task.  However, the system was not always correct. Sometimes (congruent trials) the computer was right (e.g., a subtle notification contained an unhelpful message). Sometimes (incongruent trials) the computer made a mistake (e.g., a salient notification contained an unhelpful message). Each participant experienced 2 blocks of 42 trials each. In one block the computer was correct 80% of the time, and in the other block it was correct 50% of the time. As a dependent measure we assessed the amount of system use under different levels of system reliability. Our measure of system usage was the percent of notifications opened. 2.2 Results We performed an analysis of the proportion of notifications presented that participants actually opened. A 2 (block order) x 2 (congruency) x 2 (notification style) ANOVA showed a significant interaction between block order and congruency, F(1,7)=9.56, p<0.01 (Figure 1). The half of the participants who received the 80% block first clicked on 67% of the salient notifications and 31% of the subtle notifications in that block. Moreover, in the block that followed, participants clicked on more than 65% of both types of notifications. In contrast, those participants who received the 50% congruency block of trials first only clicked on 35% of the salient and 21% of the subtle notifications in the 80% congruency block of trials that followed. In other words, the effect of the low reliability of the expert system was to significantly reduce the number of notifications opened, even when reliability of the system increased dramatically in a later block of trials. Discussion This experiment clearly showed the tradeoffs associated with high versus low system recommendation reliability and user interface presentation style. Generally, the decision to open notifications carried over from the first to the second block, despite marked differences in system reliability. When participants experienced high system reliability in the first block of trials, they were much more likely to continue to trust the system and open notifications in the second block even when the system had become unreliable. Conversely, participants first exposed to an unreliable system lost trust, and opened 20% fewer notifications throughout the rest of the experimental session, despite a dramatic improvement to reliability. These results support the findings of Maltz and Meyer (2000). The results of this experiment begin to address the effects of the critical but realistic situations in which intelligent systems make mistakes, incorrectly notifying users of the importance of incoming information. In designing intelligent notification systems, ensuring high-quality filtering from the outset is of utmost importance because once users perceive a system to be unreliable, it is very hard to win them back. First impressions cannot be overestimated in building trust with users of intelligent messaging systems. References Horvitz, E., Jacobs, A. and Hovel, D. (1999). Attention-sensitive alerting. 15th Conf. on Uncertainty and AI (UAI '99), Stockholm, Sweden. Morgan Kaufmann Publishers: San Francisco, pp. 305-13. Maltz, M. and Meyer, J. (2000). Cue utilization in a visually demanding task. In Proceedings of the IEA 2000/HFES 2000 Congress. San Diego, CA: Human Factors and Ergonomics Society, pp. 283-284. PAGE \# "'Page: '#' '" Continuing my confusion from Exp. 1, why are planned contrasts generating F values?  EMBED DeltaGraph.Document   EMBED DeltaGraph.Document  Figure 4: Proportion of notifications opened by congruency and notification type. Top, 80% congruency block first, and bottom, 50% congruency block first. 34yzklmvw $'(*+6 H!I!e!f!h!j= UV j0J!U0J! jUOJQJ6OJQJ]6j0J!<UaJ6]jUmHnHtH umH sH 5\mH sH OJQJmH sH CJOJQJaJmH sH  CJOJQJ5OJQJ\mH sH 5OJQJ\24zlm  ~ g  & F & F &    & F XD A!)"$?$*+6  !p# E^`E  & F  & F @!A!B!C!D!E!F!G!H!i!!&"'"(")"*" "$Pa$"h!i!j!!!!!!!%"*"CJ 5CJ\ j=1Uj= UV jU 9 0 00&PP/ =!S"#$%S 7= 0 00&PP/ =!S"#n$%S P D=1Dd4 CC0  # A20_tpIM"h0Dp`!0_tpIM"h2(FW582&[0x x/M`Pb{T1$$!1eE“ )#Q(P.<.X."ȋ(" .o 47dv 8s|o|̒o$wWDOVՒ.ɛ6YU_֩FK9SI߈bd$]9+ۺVzҕgck'ҒXZ ct,%'% OvJi|N+Y-8#mtݹYjҢⶮfR^J2NmZr{tJY[Ͷ|{Klq9X۸l6Ⱥ1ݿUK~<|#k~RFS(t{d.=t0s{LKu{x)Rzn)7ճ!e~[ԏ+װUQ>.~wPX4yc[(u5oq|?I-Á,W6m*'zb8Ԃ&Y@@Pq74!dڬY3AmԆÁ6,Waj@J{@-ZL1D rUV: (YZ%:b80j6mu:SC P\m۶.a&J7AڵML18JQr}zG!(Ms_ *Zi4dڡCAmÁ֔,WmLm 1Fi7r[nnL1H@rs΂:!QdڥKA}A ^rk׮S_,t!dڭ[7A]ԅÁn,Ww.b8pÐj=0SC .mTy> @PqF\K鵐jϞ=Z^ 1hAi rW^ڂ- Rz+dZXX(2V(dڻwoAS1Cr;1L1JTrԩL 1X@r_~ (Y_P70up \^y8 u;b8P/Zb8`RjBT&p3!!CjgvC=СC1',W &d?!R,dZTT$2YS!2 8gN8QPqZ#WSz5d:i$AWC r(́,W'O,9ĹL͐3<#73f@J@SL>L1Npruԩ:!)Y>:!(YN6MPg1up`5!ӧ jt'd:c Aԝ_&cY/:}㪯#w̙3wLp ,r^,fA )͇,Wx Agj>pBruقZB@E9sEÁ(} \}뭷)>1x kL} b8@o~ ;(Y;; )= Y.ZHPO34p&YK,To\"Á J3 եK jS3 yA˖-!F(Y=tSGA)-,W.\(L- Nt2dwL 1Ntrub!F) Y~6SgCS\}wu1SCWSQj nt#d{+n ~EWՎMbWC,W/_.zb4xӐՎxN34hq,+Vř ֡d:L1k Օ+W 5Lb4Fid4As)ͅ,WW^-^_ej.h m 5kֈajѠM Y~ᇂj3Ն ҿBk׮Կ2>,W?#U>,W?cSL1,W׭['`? FR(d'L}b48 uS'@R:\ݰaXeTLJgB~dL|JChTP3u>hp9!˙b41C7nԏ1hp [ >Ll#S@P\ݴia1JAc1`rq,͛7 7&7vhB[l?+Z( \/*^1 խ[ uL"RY~Wm'h9!m۶ js6 QYn߾]^<)3dcA3S 1IiOrԞL 1Gi?ruΝڏ FC(YڵKP0uhp$#!կZPG2u$h_ \ݽ{H燔N,W#q2u2hEJ_,WA}/BgQ: \ݻwXc,"JAEL]1\E*ru FQJ5(hp; ՃzTʫ; F)=Y:tHP2 h Ç )HYt^\o3,vƙ`mJkCGL 1Gi=rѣgoZb4Ji*dz1AMej*h)M!Ǐ jS6 H鍐z#hfJo,WO8!~7CS\;SC{S\7S{CCJB'O8:b4XDidz)6"J@O?1񔎇,W x @\u@_ȉ3V\ET*\|)ҥMtMtMtMtMtMtMtMtMtMtMtMtMts;oUK2mY-1Ք:jjsJ^ԫ!(ڜ70~O ڜb BJ !)YB~"jsJ1OxjsJgx~k&3!)93!.6r.跶Rڜ[:A jsJ` ~7iu[:sJ:}ooPYmN@[d9%-ZVwJCVS;SC[)YmNɁLot,d9%2u,DAVS9>oMې|oC[)]YmNL]o  !"#Z&@*+,-./0123456789:;<=>?BCDEFGHIJKLMNOPQRSTUVWX[\]^_`abcdefghijklmnopqrstuvwxyz{|}~Root Entry  FHi(Data ;bWordDocument z6ObjectPooliHi_1036167297 R`iiOle ObjInfoContents)-  Oh+'0 8D ` l x Interact 2001 Word TemplatenteMEGD - 00 &z8|RV f ) H ` l m  F %` &r  :     H -f "&:Ex3Z"<o      ??????????????????3"(????????????????????????????????????^"&Brst(,D\   >"&2v"&N n o n oD  noD  no@  loProportion Notif. Opened^"&BPP@ p@f"&Fpp@ 0p@f"&Fpj"&p????IA@????p@@@??@ @PA@@@H????/0ExK 2,QBu. # ! o(8,<04$rstuvw+:)0O   0u"\   u"\@%}@%} ,u( XNNK@`@` ٯٯٯٯ    First block ( XSSXHbX(r&h82xHGroup 1SalientSubtleSalSEMSubSEM80/20Pc}h??P>??Prwq??Pܰ??50/50 P9/??PN)x9??P4Bx¾??P"?I??Group 2 SalientSubtleSalSEM SubSEM50/50 P??PI$I$??P2I??!PPg{z??80/20 P$I$I??P$I$I??P ??"P5_??!RJx 6 h N pVN >x6 `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`     } }   `FJ\j x00@`     } }   `FJ\j x00@`         }  }    `FJ\j x00@`     } }   `FJ\j x00@`     } }   `RVhv 000@`      } } }   `FJ\j x00@`     } }   `RVhv 000@`           } } }   `RVhv 000@`      } } }   `FJ\j x00@`}}2  `RVhv 000@`           } } }   `FJ\j x00@`}} `RVhv 000@` $+ %+ %+ %+ %+ %} %} %} $$ $$ `RVhv 000@` ! ! ! ! ! } } }   `RVhv 000@`      } } }   `RVhv 000@`       } } }    `RVhv 000@`      } } }   `RVhv 000@`%,&,&,&,&,&}&}&}%%%% `RVhv 000@`"""""}}} `FJ\j x00@`}} `FJ\j x00@`    }} `FJ\j x00@`}} `FJ\j x00@`     } }    `FJ\j x00@`}}2  `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`}}ArialArialTimes New RomanArial_1036167356 R`(i(iOle ObjInfo ContentsA9.MEGDM. 11 &z8|RV f ) H ` l m  F %` &r  :     H -f "&:Ex3Z"<oo ??????????????????3"(????????????????????????????????????^"&Brst(,D\   >"&2v"&N n o n oD  noD  no@  loProportion Notif. Opened^"&BPP@ p@f"&Fpp@ 0p@f"&Fpj"&p????Y A@????p@@@??@ @PA@@@H????ExK ,H=u # ! o(8,<04$rstuvwv VLL u:)0OLL 1u"\.@ .@   u"\.@&@~.@&@~ u( XNNK@`@` ٯٯٯٯ    First block ( XSSXHbX(r&h82xHGroup 1SalientSubtleSalSEMSubSEM80/20Pc}h??P>??Prwq??Pܰ??50/50 P9/??PN)x9??P4Bx¾??P"?I??Group 2 SalientSubtleSalSEM SubSEM50/50 P??PI$I$??P2I??!PPg{z??80/20 P$I$I??P$I$I??P ??"P5_??!RJx 6 h N pVN >x6 `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`     } }   `FJ\j x00@`     } }   `FJ\j x00@`         }  }    `FJ\j x00@`     } }   `FJ\j x00@`     } }   `RVhv 000@`      } } }   `FJ\j x00@`     } }   `RVhv 000@`           } } }   `RVhv 000@`      } } }   `FJ\j x00@`}}2  `RVhv 000@`           } } }   `FJ\j x00@`}} `RVhv 000@` $+ %+ %+ %+ %+ %} %} %} $$ $$ `RVhv 000@` ! ! ! ! ! } } }   `RVhv 000@`      } } }   `RVhv 000@`       } } }    `RVhv 000@`      } } }   `RVhv 000@`%,&,&,&,&,&}&}&}%%%% `RVhv 000@`"""""}}} `FJ\j x00@`}} `FJ\j x00@`    }} `FJ\j x00@`}} `FJ\j x00@`     } }    `FJ\j x00@`}}2  `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`}} `FJ\j x00@`}}ArialArialTimes New RomanArialt?d9%3u?DUQݖc✒N߸Z#zփ6d=փ跚RڜM߲)!)i3ՆPڜ}NpjsJgp~k"!)9!)}ڜ3u~k%+!)+!n6vnNQz ڜz ߪ^887IwDՈFlFVd9%CL A[Qzd9%ocm JAVSrSA[OPd9%`4JAVSrSA[(]YmNEL]oE)BVS2(DuC Yg,DoAV[L} ҏ }&#~otda~+9nKGqNIo\MrDUҺ>^u!lJ!}"=V>>;|~J);zDwJYg!ȪsJW#+9ܲP/S;٩URKMtMtMtMtMtMtMtMtMtMtMtMtMt"s;-ѰJ9֎ yh6i ]Zl9 ^qTG2(̀6dS3 FB(mYSAzRڜ=b$ BV{yB)-3b$4攜#.nK!FB) YmNL 1:F1jǘz b$tyZ<:}i ]O攼CZS;sk ud9%1Hh !d@XJBVSr,SByjG3y;YmNwHh l`J@VSS@.nTl9ijG 1CjsJaZJBVSr-SBvSڜb$^l9ݍttGRZڜuZb$Mi6d9% 1ʧ4ڜL͇ RZYmNBB(-6dS FB(Ym L1zW!2U JW@VSrSW@Q ڜۘ b$tғ<ԓ#:Ŗ867pH )و FBm)m YmNɶLm 1*ڜL- t0d9%3u0Hh )9 FB)YmNL1ZBjsJ.a&J7AVSrS7APzڜGzb$tYfd9%3u>HJ?6'Lb$}}#-/SmNIo\#zփ6d=փ R YmN\B:Rڜb$twAVS.1zG )S Mtd9%'1uHh6!8!FBk(]YM]5#]6. -籫ڜN߸rHJJ6Lb$ԘƐl#6d{ d9%{1Hh )9 FBJ#攌051zW '~tdNv*an.jsӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{ӵ7]{7εV'yb֦ަs){>.*[|(ukSW\lRjKUu=N.mq2znLjugtԆ{BRgP4lNrߴI\Vn֛AlielEnAiǶAliel=G[ͳ{f<]zlNm=g-74Ms{lLs{lLk[ʋhazXVGXx)RjR^l=GX+b="֣N=bKyRSSŖb=:zt)֣cY׉Y'n8OY7-u{-Kw{n"_cZT׿״TGG3!r{̈Ery#ۣz w뽑5cQ[w?\9gkylNr^Jcσ8W,[*=$'%%]O]Mɹ%|W-Ww;:ؙxqG_AˌwVMWrvQ ol{QskĞ w vNfFΏHV~8I3j.; _C\}nۥ}/]vfcL{/cue{]u. E[:Ua:}wҭ7'Rcn=!']y_|%>BON󵾴Es}{+TO3٬i^仲ۀ$[n5=~=5jǯǯǯǯǯǯ^\Ts?tUs xTB=g*nVa*/yvS+Oe?!oUX HWJw!j|V T'=~=~=S_oe?ǟU 馛nUR\0s:*oUҝ<&dw=~=~}G__r*I>]B(QeT AU*V|:џ?_zzztMK߿_5_1+8x=~ ?G RȶNj|'/=1~qrezz/p]e.q8O|͊phWOK?υ_??8M7ݪƥC= a2I]O2PC彐?n馛n馛nvOrz?z^B{wAC_zzۅ yz{ys__vAǯǯOЯo?A/*8~ /`zUcߓt=0~t_zUX] J??g>tW3/릛n馛n馛nUVXF^e>_aO/?x]D]I[_r.[V=~=~=1Y߿Oee?~NL")?V_W\7tMϋ! +o.?Tz=~=~=/ zF}-%]2Lv!_2S*pLGȕGHo?2J1c8?%JSU?>x^ չTϢɩnv~ܫA+Ł)~HKOaj DL4\mڴ25mSYAMgj:D̤4\m֬f25mfS YAAfj6D̡4\mѢ05mR YjJPs 6Q \,KP1DlNisrM6ڜ!fKJ[Bm۶ԖLm 6-J-r]vj1Ղhm(mYo^P0 Dۼқ |A7Amۂ|ف-"mv#dڹsgAԎm .]j'vh](YvUP0 DJiWr[nڕ]!fwJCݻwL6{P\ѣ`jy;Cz;SohaJÐjϞ=50D,\ի2m7dZXX(mE]j޽.6R\;/SB,WnAms ~ hQzdڿAAA,W  :mtd:p@A!ms(C!_P2u(D,\DC!C!>6ruСSh#) Y6LPG2u$DMhrHPG3u4DCr1L61rǘD|' #FL}mSZ Y5JPZ 6Iruт$Sh(Y3FP'0uD7,Wǎ+f!$J'A?Nb$4OCƍԧ4DۜBrXP0u DۜJTr'ԩL 6_ruS_h/R"dSO L}mΠtd:qDAmeJ_,W'M$/3e*B'OW*D|!gyFP_gms6!)SlΆhs(YN:UP0uDۜGZ-Vf*%Sh)YpL6wR\}7u'SwBͯ)\-N_3k @so DG>rc>y҃ԃL=6Sz\7ofayңjbW=ԣm8! qhS=dZUgm z f %… %m3)mt3QrjWuդ4GV,W/^,ՙZm֠dZhUk0DۼK w}WP/a%mRJ/,WLmrRrZL6M!GNkm֦6d|rAm\-<檿eo!敔^ YXBPdm*J,WUL mC+W3m ǎZ f}JCWL6H!5kGـ~( mPY]VPS6(M,W?#AMcjDL4\5m1!u jc6hٔfC|f35m6 d~zAm&m3\ruÆ \iRjB~L5!fsJChTP39DlEi+jj+hdqFAjA!r>SCm&Jo,W7m$71&ٞ.mSYn޼YP6ofru˖-z3Soh)Y~ڑ!fgJ;C[nL 6P\/ S@ͿPru۶mmv;d}vAm6Jo,Ww!16y;C_}!f/J{A;w^L6 )-,Ww%L-h}(Y~ׂڇ} ]Y޽[Pb]m ={j?h)Y~7ڟ!潔 YݻWPem>J,W'1>9u0SC!,W8 C:mtdzAAam"rСCZ"9Çu8SCG(}\o>6GQ: \=r䈠b(9ѐѣGu4SGCG)}\=v옠>G!c>Y?~\Pccms :mSZ Y8qBPZ 6#F NP#L@ N,W{A ms"!Փ'O DNh(Y:uJP'1uD|g ӧO 3L}mNt dWP0u D| I> YU›v*gg FXi=Щ.9U7w)3߅W]tMtMtMtMtMtMtMtMtMtMtMtMt.{+3͕ Ǽy n g#[Xث-m;^loSث=wX{t3h0Dd4 CC0  # A2z0oFk;V01p`!N0oFk;25FA82&0x tU7NH*epT$p1pPAt" 0r """ aDdk1(F4{T'j$i([M]UOMB^ Pg`9jIpjOK|"f鉃Oo\RuC ϩD#0hUm] 2%NGu03Og)1!Qж4%'zrb,,%$:N(gJ {{vYshiA[wia[W]7%\ws-=JS֛ݖ6*BZX*q|6gNq>zIJ3Wq?g+_Jl]>nn c=p{Lp{Lp{LX[qcۣsۣsۣszn)H. ⼤.as~='wTJ߇͛7~w?u@n1æM`Vlq}?''L! NV$uS'@ )Y)ә:b̧4ZȐ|C )YZ%bE.j5++KRw1uİYJi)dڲeKI-ej)İyFFn(*TozF#F6V۴i#blNisjm۶ڜ!ͶVsrr$-SB ]) Yo^R2+İه>j$S@ C(Yp :C g)}Zر>g!(} Zԩ ٔΆV;w,:b\A jK.+ ͭnVv*[b,Z[$EfbTQT{P[8bL4Z$5e^YJeL b̠4Zѣf05bؼ!;CRgf.jϞ=%5f?JAVwucj?a)JVI}OA c) YGR2u,İ9ҩ}'S:b؜O|jo߾:!͏)Z}%c~ 1ltdCI6PzZ߿`a<TA0`:}+z#͆6V $ bؼҫ!GyDRff+J[AVVLm1lv3d:dIf/J{AVC^L1ltdK 6GP:Z}'%uSG@ )YIDN6C ÇKas1!U۶%u1SC ) Y1BR73u3İYHi!d:rHI-dj!İyÐQ$0SC ge:UPG-N LG Sz1dKLbL4Z3f05b̡4Z;v05bNiwjuܸqڝ!)Z?~!͡V'L C:b| Չ'JsL}b؜LdjW^L 1lΥt.dꫯJ\΅6WQ Z4ib*as _]R1uİҽ)S$u/SB IYcoԩSJӷBMrİy>CVӦMz>İyW@VӧO+zİiQjAVZL )Z1c!0aju֬YfjbOijwߕL1ltd:w\Ia8JAV͓qL1lNtd:|IiJ@V|.`aJ?V,X 0as';!|IԝOY.^XRbOiٹ*K.TozZ# AjudjblJiSjuʕڔM!֔VW^-bBijr re;UO+qRETʖZTղZ۱ƤkokokokokokokokokokokokokoVko%=[݈SWEM}nuÍ njZ)uspr̙󶜩!F͔nUy3S7CZ;) YΞ=[dNQb^닙Z 1jitUP̙#N N GZ (m;H6Q?SAZM(mY{ڄM F,J {bjĨu=CVVwJCڝ!Fޔ=7S{CZ(Y~5 S:wL1jt dpBI1$J'A~? F΄V-Z$3:bZD""1uĨu%K$uSAZ@wlRQ"juٲerEJ@wsG F38RuŊ#F68!SBZWRz%dj*IWBZ&&dGg&SMQ+ju͚5U0@Z](Y~Gڅ] FV׮]+=bz҇!Տ?XRf>Y[NR`ֳ> Y~''$,Ĩ2/CVׯԗ2Ĩ5 Ou1u:Ĩ@V7nĨՐM$u5SWCZ[(Yn޼YR0u ĨUHi!dF= BQ@Zu37Σ ꧟~*N n#F(=Z$<1j]J饐-[$R^ 1j]K鵐֭[%Z^ 1j5dgLm 1jDMm$&1jN_|!3vQ/}!ՂIԾJ@V_~6FP:Zݾ}\eQEJ_Vw!/2EQMJ߄V+I}oBZs) Yܹӣ®@Z(]Y~ג FMnVIMNJwBVv풯0u'ĨULi1d]Z 1j%fm4GqB3xY,GZgSz6dwIL=bjDi#juu6PZݳg|݅M F,J ufAZ(YIjvSZ.L1j7dw^ S{CZ(Y۷OA0JAV?G: bCjX1$J'AV0uĨ5ҙ.L 1jSY}L͇Q?uSAZ@V?|݅VEjII|݅EaJCV?'Mq jii:}+ӳ1j5!dz! SBZWRz%dza S)MV9"LM:jU39:^1ju sۍzٱsKT+90 WTղ_MtMtMtMtMtMtMtMtMtMtMtMtMtvĠXKZ\Եb+[!Sr+SB o)9%e SS@ :<27לN AqgA7YL= bP?A7䟘'AјƐ)٘!E:)tA4ߜ! E'J;A7d'vB7d.Ss!Ž ߜ2^A1A)9 Ey)1(FS:9%G3u4ĠHD攜ԉ-J߂ĠK\uԹb)K!`)SB 뙺bPPZg Z1(PaAq҃}z SB AsJ:}+ԺَRz.dH?B )ώ_ԋ!EJ@)&Lm1(,J->m1Ղ(m'1ĠBi 2}n:$:0 WTղ%MtMtMtMtMtMtMtMtMtMtMtMtMt:vdsJgj}PJ@7c6 ?B ]DE)yS/ ]A)yS ]C5)y S )i2Մ %dsJdjKP9)PGJ;B7dGv L͐)y3So R ߜLͅ E])yS /dsJej__)+dsJ tdsJaPy)<3>ߜ0PdsJF MtdsJN`$J'A7$N E[)S߂ ͤt&dsJdLߜ3}"JA7". tdsJb*zJC7z mt dsJnavJC7vn RZߜL- tdsJc>P %)YPBh*_w4W|?|k']:3]/)X4 ǚ3c=[(Q>S2&,sFX3Kɉ[[n;%celiUlnm-ۼE՞~feN.[Xߺ+6o,_Tl)/#nd+3bKyY紩s}6eN^4|)9qp9m\ۣuzwYڸK"^Z#/푗*5[`;j۳9gĖڥNYYnY9vz4hk>8nu/>@=;\KjOvۙho3wKU{]u.oxo}T}je[O]Tje[OkW'\yz9@ם؝KJ8;fӲ39!pAY?oGul߬׬Zk_=~=~=~=~=~=~=rj:cڟSWãnjp=W9tMjV _oϳ#\_j=~}\kEkt-8 qRjgTzǯǯ_s_j-x?ׂ/tӭv\џK?\Gv׸xL}ǯǯ?{B=)ӧ%j͵Zp!\ŪY?'K____?nsk!qM<: ǯrׂ=~=h ׎9*?_;Ưۉ=N?B__7tke-85vYQO=~=~=~s՟ǟV馛nR0yH\ϤN'__֚]q￵^HџM7tM7tM7t;'׾׎b z/?{wACл!^/=~=~tdkztW3O릛n馛n馛nVXF^k>_a?x߻\ku??l +>oժ~ZS kkgMe?5q593| F_}D_)[-nz^\=yuVXsqu>UKu(7oR߅{.SsOn'EZs9\Wu\~j~ 7׈_=~=Zs@/&^WBru-5(s}rn3AP=~=q`L_^__rsbw#s(twx!5ŵ(U@<џ???_Ư?R.ϟT]>N[OtM_!5!~| ::]Я ժ'Mv_7tM7tM7tM7tM7tM7tM7tM7tS:@ K6CR[%]0kR"%y7o|kRwKNtc[ IuT[YIL}-(̓VMӔg b Ֆ-[J(hєV:!"BiZmӦFh(}Zm۶ b ՜Iqm1j%uCjgϞ!S?h|J!ջKR)] Ys=!b >}H2.hV>I]mՐj߾}%u5SWCG~Y>S?huVz!I]umj%u=SC&J7AV MLQJA$5(D[lt d#Hnh)Z)0(Z˓BBnJwCVÇL EAVmKjS b/{!#FH^hbJ!Ց#GJj1S! QFIhJK ѣGKj SK ڢRj矗RBJ@Vcƌ#L=VHԱcJӷB 9-PZZ7naj0(5 KT-NTju„ z*SOhևV'N(Z-Π jW^3zD[I限ꫯ*g2Lh@ijuҤIڀ "dj_d&CYN2ERgmJYN:UR?@Ņ^YN6MR/dm4YN>]RL BE^Y;zS/hK(Z1c^K ϔZ5kh)Z}w%r^)m YΝ;WR31D[\EUy$*^M)m YΟ?_R2)D[4dHj36hJS  Hj SS "4j5??_RӘf@V/ f@EY.]TR-ZmMi6d|rIfj6D[drJImVmц6ի% S@E[JBVepFya-i/b(*aeQ",TʖjTղ۱TfJ7ófY{3HRݬ= \NI5Vc;n*Wu^WUOn R瑚>b̙3+eL5!ڢ- ڂ- ",juْ,hIiKj;djKhMikjuΜ9ښ!"DiZ! "@E{JCVgj{!Ůz=Sh)Z}%F(YV~wNL) YV~wL 7Sz3dJL-Q Z|ݘ -nVju…z+SohPjшhV-Z$=-j1; ڢ %KHj/h)Z|w3nCijuٲeڇ} >JV+}L-juŊSh~V+j?hVWZ%-P:Z|:-F 5kHߘ7L`j>L R(dvZI}Bc>Y~ǒSh')}Z]n>'!"J#j4S#m1ұc:-Q:ZO%uSAJ'@V}N`HDju˖-:!_ Zݺuh(} Z%5S:dm6I}CJ@V_|Naxҷ ՂI}oA۔ Y~嗒6S߆hw(}Zݾ}w b3 ;vH ΀hٔΆV+Im19Ν;%uS@Sju߾})S?h-nVI-mmjqqnc6(Zݿ0-vPZ=p`INjԝL (ٟ ڢBj~BBJ@V%%%{ ڢ"jZ"G>jTR1uD[SZ Y:tHRZ ?R#dzaI?BE %#G$%mqCQIb!8Bjr/I *gǺ JzS +[:RiǾ /ItMtMtMtMtMtMtMtMtMtMtMtMtގ.@> Normal$a$PJ_HmH nHsH tHd@d Heading 12$ & F hx@&^h`a$5CJ\aJ`@` Heading 2-$ & F hPx@&^`P5CJ\aJ:: Heading 3$@&6OJQJ]^^ Heading 6"$$P5$7$8$9D@&H$a$5PJWvSK\tH <A@< Default Paragraph Font,+, Endnote Text, @, Footer  !,@, Header  !:O": title$XDa$5CJ$\aJ$8O28 Author $a$5CJ$\aJ$4OB4 Address $xa$CJaJ$OR$ Abstract4Ob4 Body `OJQJ0r0 Bullet^`\O\ Reference0HeadingK^`K5CJ\aJ,Oa, 1st para `<B< Body Text$a$CJaJmH sH .U@. Hyperlink >*B*ph<C< Body Text IndentCJaJ&)@& Page Number>Z> Plain Text$a$OJQJmH sH >V@> FollowedHyperlink >*B* phJOJ Reference :^`:CJOJQJaJ:'@: Comment ReferenceCJaJH@"H Comment Text"P5$7$8$9DH$PJtH 8&@18 Footnote ReferenceH*ZBZ Footnote Text&$pP5$7$8$9DH$^`pPJtH @RR@ Body Text Indent 2 %r^rb References& & FP5$7$8$9D>T.H$Tf^`PJtH "W@q" Strong5\@T@ Block Text( ]^6*@6 Endnote ReferenceH**&marycz'*mor*6;64zlm~    g$?$*+6@ABCDEFGHi&'(+00000000`000000000 0 0 0 0 0 0 00 000000 0 0@0@0@0@0@0@0@0@0@0@0@0@0@0 0@"0 0000000000"0000 h!*" *")"r!$&(EG::8&'@&(  h & S &" B S  ?*&kD6=A+FxA+3333   [\]]hh$$01??$$YZS_@AAB(+Scott LeeTiernan]C:\WINDOWS\Application Data\Microsoft\Word\AutoRecovery save of boggle_interact2001_short.asdScott LeeTiernan=G:\notifications\boggle2k\paper\boggle_interact2001_short.docScott LeeTiernan=G:\notifications\boggle2k\paper\boggle_interact2001_short.docScott LeeTiernan=G:\notifications\boggle2k\paper\boggle_interact2001_short.docScott LeeTiernan>G:\notifications\boggle2k\paper\boggle_interact2001_shorta.docScott LeeTiernan>G:\notifications\boggle2k\paper\boggle_interact2001_shorta.docScott LeeTiernan^C:\WINDOWS\Application Data\Microsoft\Word\AutoRecovery save of boggle_interact2001_shorta.asdScott LeeTiernan>G:\notifications\boggle2k\paper\boggle_interact2001_shorta.docScott LeeTiernan>G:\notifications\boggle2k\paper\boggle_interact2001_shorta.docScott LeeTiernan>G:\notifications\boggle2k\paper\boggle_interact2001_shorta.doc C!,$Qz7/%)3,cq^wRvRmؿhh^h`o(0^`0o(.0^`0o(..0^`0o(... 0^`0o( .... 88^8`o( ..... 88^8`o( ...... `^``o(....... `^``o(........hh^h`56CJOJQJ^JaJo(.1hh^h`P^`P^`...x^`x.... ^` .....  X ^ `X ......   ^ `....... 8^`8........ `^``.........^`o(.?^`?o(?^`?o(.0^`0o(..0^`0o(... 0^`0o( .... 88^8`o( ..... 88^8`o( ...... `^``o(....... `^``o(........0^`0o(0^`0o(.0^`0o(..0^`0o(... 0^`0o( .... 88^8`o( ..... 88^8`o( ...... `^``o(....... `^``o(........@p^`p.k^`ko(k^`ko(.0^`0o(..0^`0o(... 0^`0o( .... 88^8`o( ..... 88^8`o( ...... `^``o(....... `^``o(........0^`0o(0^`0o(.0^`0o(..0^`0o(... 0^`0o( .... 88^8`o( ..... 88^8`o( ...... `^``o(....... `^``o(........0^`0o(0^`0o(.0^`0o(..0^`0o(... 0^`0o( .... 88^8`o( ..... 88^8`o( ...... `^``o(....... `^``o(........ q^$QwRv%)3C!7/ 3y ABCDEFH&+@HP LaserJet 5Si/5Si MX PS\\Shodaserver\hp5simxADOBEPS4HP LaserJet 5Si/5Si MX PSHP LaserJet 5Si/5Si MX PSw odXXHlR}0RdCustom page 1BBCustom page 2BBCustom page 3BBHP LaserJet 5Si/5Si MX PSw odXXHlR}0RdCustom page 1BBCustom page 2BBCustom page 3BBd*`@UnknownG:Times New Roman5Symbol3& :Arial3TimesG MS Mincho-3 fg?5 :Courier New#CRCRK ( 2!0d2QInteract 2001 Word TemplateI2001 Program CommitteeScott LeeTiernan