Chapter 5 Moment Generating Functions | bookdown-demo.knit

Example 5.1 (Exponential MGF). First, we'll work on applying Property 6.3: actually finding the moments of a distribution. We'll start with a distribution ... Probability! Preface Foreword Aboutthisbook Features Acknowledgements Dedication R GettingStartedinR Functions Looping Forloop WhileLoop Graphics Plot() PlottingTechniques Glossary Practice Problems 1Counting Sets NaiveProbability SamplingTable StoryProofs Symmetry Practice Problems BHProblems 2ConditionalProbability ConditionalProbability TheBirthdayProblem MontyHall Gambler’sRuin IntroductiontoRandomVariables PropertiesofRandomVariables Binomial Practice Problems BHProblems 3DiscreteRandomVariables RandomVariableRecap Bernoulli Geometric FirstSuccess NegativeBinomial Poisson Hypergeometric Expectation,IndicatorsandMemorylessness Expectation Indicators Memorylessness Practice Problems BHProblems 4ContinuousRandomVariables Discretevs. Continuous LoTUS Uniform Universality Normal Exponential Practice Problems BHProblems 5MomentGeneratingFunctions MomentsandTaylorSeries MGFProperties UsingMGFs Practice Problems BHProblems 6JointDistributions JointandConditionalDistributions 2-DLoTUS Multinomial ChickenandEgg Practice Problems BHProblems 7CovarianceandCorrelation Covariance Correlation Transformations Convolutions MVN Practice Problems BHProblems 8BetaandGamma Beta Priors OrderStatistics Gamma BetaandGamma PatternIntegration PoissonProcess Practice Problems BHProblems 9LimitTheoremsandConditionalExpectation LawofLargeNumbers CentralLimitTheorem ConditionalExpectation AdamandEve Practice Problems BHProblems 10MarkovChains IntroductiontoMarkovChains CharacteristicsofMarkovChains StationaryDistributions HiddenMarkovModels Practice Problems BHProblems Epilogue AcademicResources StatisticsintheWorld ProbabilityPlaylist FinalWords References Publishedwithbookdown Chapter5MomentGeneratingFunctions “Statisticsmaybedull,butithasitsmoments”-Unknown WearecurrentlyintheprocessofeditingProbability!andwelcomeyourinput.Ifyouseeanytypos,potentialeditsorchangesinthisChapter,pleasenotethemhere. Motivation We’llnowtakeadeeperlookatStatistical‘moments’(whichwewillsoondefine)viaMomentGeneratingFunctions(MGFsforshort).MGFsareusuallyrankedamongthemoredifficultconceptsforstudents(thisispartlywhywededicateanentirechaptertothem)sotaketimetonotonlyunderstandtheirstructurebutalsowhytheyareimportant.Despitethesteeplearningcurve,MGFscanbeprettypowerfulwhenharnessedcorrectly. MomentsandTaylorSeries BeforewediveheadfirstintoMGFs,weshouldformallydefinestatistical‘moments’andevenfreshenuponourTaylorSeries,asthesewillproveespeciallyhandywhendealingwithMGFs(andhavelikelyalreadyprovenhandyinthePMFproblemsofpreviouschapters). The\(k^{th}\)momentofarandomvariable\(X\)isgivenby\(E(X^k)\).The‘firstmoment,’then,(when\(k=1\))isjust\(E(X^1)=E(X)\),orthemeanof\(X\).Thismaysoundlikethestartofapattern;wealwaysfocusonfindingthemeanandthenthevariance,soitsoundslikethesecondmomentisthevariance.However,thesecondmomentisbydefinition\(E(X^2)\)(plug\(k=2\)into\(E(X^k)\)).Weknowthat\(E(X^2)\)isnotquitetheVarianceof\(X\),butitcanbeusedtofindtheVariance.Thatis,\(Var(X)=E(X^2)-E(X)^2\),ortheVarianceequalsthesecondmomentminusthesquareofthefirstmoment(recallhowLoTUScanbeusedtofindbothexpectationandvariance). So,essentially,themomentsofthedistributionaretheseexpectationsoftherandomvariabletointegerpowers,andoftentheyhelptogivevaluableinformationabouttherandomvariableitself(we’vealreadyseenhowmomentsrelatetothemeanandvariance).Sincewe’resuchmastersofLoTUS,wewouldbecomfortablefindinganyspecificmomentfor\(k>0\),intheory:justmultiply\(x^k\),thefunctionintheexpectationoperator,bythePDForPMFof\(X\)andintegrateorsumoverthesupport(dependingoniftherandomvariableiscontinuousordiscrete).Thiscouldtakealotofwork,though(andwehavetodoaseparateintegral/sumforeachdistinctmoment),andit’sherethatMGFsarereallyvaluable.Afterall,it’sinthename:MGFsarefunctionsthatgenerateourmomentssowedon’thavetodoLoTUSoverandover. Let’snowbrieflyfreshenuponourTaylorSeries,becausetheyareprobablyasrustyastheyareimportant.WewillseeTaylorSeriesarecloselyrelatedtocalculatingmomentsviaanMGF,soit’sveryimportantnowtofeelcomfortableworkingwiththeseseries.Herearethechiefexamplesthatwillbeusefulinourtoolbox. TheExponentialseries: \[e^x=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+...=\sum_{n=0}^{\infty}\frac{x^n}{n!}\] Geometricseries: \[\frac{1}{1-x}=1+x+x^2+x^3+...=\sum_{n=0}^{\infty}x^n\] \[\frac{1}{(1-x)^2}=1+2x+3x^2+4x^3+...=\sum_{n=0}^{\infty}nx^{n-1}\] Theseholdfor\(00\)(thisconstraintisfine,becausewedon’tneedtoconsidera‘negativemoment’). Voila!ThisisthefirstMGFwe’veactuallyseen.Prettysimple,no?Noticeherethat\(\lambda\)isfixed,and\(t\)iswhatwearegoingtoplugintostarttogetthedifferentmoments(moreaboutthatinasecond).Let’sconfirmthatthisisinfacttheMGFwithasimulationinR.Wewillplottheanalyticalresultabove,aswellasempiricalestimatesof\(E(e^{tx})\). #replicate set.seed(110) sims=1000 #defineasimpleparameter lambda=5 #generatether.v. X=rexp(sims,lambda) #definet(inanintervalnear0) t=seq(from=0,to=2,length.out=10) #calculatetheempiricalandanalyticalMGFs MGF.a=sapply(t,function(t)lambda/(lambda-t)) MGF.e=sapply(t,function(t)mean(exp(t*X))) #plottheMGFs,shouldmatch plot(t,MGF.e,main="ExponentialMGF",xlab="t", ylab="",type="l",col="black",lwd=3) lines(t,MGF.a,type="p",pch=16, lwd=3,col="red") legend("topleft",legend=c("AnalyticalMGF","EmpiricalMGF"), lty=c(1,1),lwd=c(2.5,2.5), col=c("red","black")) Now,whatcanwedowiththisMGF?Remember,forourpurposesthereareessentiallytwowaystogetthemomentsfromtheMGF:first,youcantakethe\(n^{th}\)derivativeandplugin0forthe\(n^{th}\)moment.Second,youcouldfindthecoefficientfor\(\frac{t^n}{n!}\)intheTaylorSeriesexpansion. Thatsecondonesoundsalittlevague,solet’sstartwiththederivatives.Saythatwewantthefirstmoment,orthemean.We’lljusttakethederivativewithrespecttothedummyvariable\(t\)andplugin0for\(t\)aftertakingthederivative(rememberhowwesaidthat\(t\)wasa‘dummy’variablethatwouldkeeptrackofthemoments?Thisiswhatwasmeant): \[M'(t)=\frac{\lambda}{(\lambda-t)^2},\;\;\;E(X)=M'(0)=\frac{\lambda}{\lambda^2}=\frac{1}{\lambda}\] Incredibly,thisisthecorrectmeanofan\(Expo(\lambda)\)randomvariable.Let’sdothevariancenow;wealreadyhavethefirstmoment,butweneedthesecondmoment\(E(X^2)\)aswell.Wecanjustderiveagainandplugin\(t=0\)tofindthesecondmoment: \[M''(t)=\frac{2\lambda}{(\lambda-t)^3},\;\;\;E(X^2)=M''(0)=\frac{2\lambda}{\lambda^3}=\frac{2}{\lambda^2}\] Nowthatwehaveoursecondmoment,wecaneasilyfindthevariance(remember,thesecondmomentisnotthevariance,that’sacommonmisconception;wehavetotakethesecondmomentminusthesquareofthefirstmoment,bythedefinitionofvariance): \[Var(X)=E(X^2)-E(X)^2=\frac{2}{\lambda^2}-\frac{1}{\lambda^2}=\frac{1}{\lambda^2}\] ThisexactlymatcheswhatwealreadyknowisthevariancefortheExponential. However,thisseemsalittletedious:weneedtocalculateanincreasinglycomplexderivative,justtogetonenewmomenteachtime.It’squiteusefulthattheMGFhasthisproperty,butitdoesnotseemefficient(althoughitisprobablymoreefficientthanemployingLoTUSoverandoveragain,sincederivationiseasierthanintegrationingeneral).Let’strythesecondmethodofgeneratingmomentsfromtheMGF:findingthecoefficientof\(\frac{t^n}{n!}\)intheinfiniteseries. Thisseemsambiguousuntilyouactuallydoit.WefoundthattheMGFoftheExponentialis\(\frac{\lambda}{\lambda-t}\).Doesthatlooklikeanyoftheserieswejusttalkedabout?Well,itdoeskindoflooklikeaGeometricSeries,butwehave\(\lambda\)insteadof1.That’snotabigdeal,though,becausewecanrecallourusefultrickandlet\(\lambda=1\),whichisjustan\(Expo(1)\)(recall‘scalinganExponential’fromChapter4).Let’strytoworkwiththat,then,since,likewehaveshownearlier,it’sprettyeasytoconvertintoanyotherExponentialrandomvariable. Pluggingin\(\lambda=1\),weseethattheMGFforan\(Expo(1)\)is\(\frac{1}{1-t}\),whichisdefinitelyaseriesthatwehaveseen.Infact,expandedout,that’sjust\(\sum_{n=0}^{\infty}t^n\)(seeabove). So,wehave\(t^n\)inthesum,butweknowfromourlistofthe‘properties’oftheMGFthatweneedthecoefficientnotof\(t^n\)butof\(\frac{t^n}{n!}\).Canwefixthat?Sure,ifwejustmultiplyby\(\frac{n!}{n!}\),wewillget\(n!\frac{t^n}{n!}\),andthen,clearly,thecoefficientof\(\frac{t^n}{n!}\)is\(n!\)(thisisatrickystep,somakesurethatyougothroughituntilitmakessense).So,themomentsoftheExponentialdistributionaregivenby\((n!)\).Thatis,\(E(X^n)=n!\),ingeneral. Let’sdoaquickcheck.Weknowthatthemeanandthevarianceofan\(Expo(1)\)areboth1,sothisshouldmatchupwithwhatwe’vejustfound.Well,themeanisfirstmoment,andplugging\(n=1\)into\(n!\),weget1,sothatchecksout.Thevarianceisthesecondmomentminusthefirstmomentsquared,andthesecondmomentis\(n=2\)pluggedinto\(n!\),or2.So,wejustdo\(2-1^2=1\),whichisthevariance,andthischecksout. WecouldalwaysconvertbacktoanyExponentialdistribution\(X\simExpo(\lambda)\).Remember,if\(\lambdaX=Y\),then\(Y\simExpo(1)\),andwealreadyhaveaverygoodwaytofindthemomentsforthisdistribution.Wecouldturnthisintofindingthemomentsof\(X\)veryeasily: \[(\lambdaX)^n=Y^n\rightarrow\lambda^nX^n=Y^n\rightarrowX^n=\frac{Y^n}{\lambda^n}\] Andthenwecouldjusttaketheexpectedvalueofbothsides: \[E(X^n)=E(\frac{Y^n}{\lambda^n})=\frac{E(Y^n)}{\lambda^n}\] Remember,\(E(X^n)\)isjustthe\(n^{th}\)moment,andweknowthatthe\(n^{th}\)momentof\(Y\)isjust\(n!\),sowecanplugin\(n!\)for\(E(Y^n)\): \[E(X^n)=\frac{n!}{\lambda^n}\] Andwehaveanice,closedformwaytofindthemomentsforanyExponentialdistribution,whichismuchfasterandeasierthancomputingtheLoTUSintegraleverytime. Additionally,it’sclearthatthisisevenamuchcleanerand,inthelongrun,easierwaytofindmomentsthanusingtheMGFtotakederivativesoverandoveragain.Insteadoftaking10derivatives,forexample,wecouldjustplugin\(n=10\)tothatexpressionthatwegot;it’smuchsimpler.So,rememberthatoftenareallynicewaytogetmomentsisjusttoexpandtheMGFintoasumandfindthecoefficientof\(\frac{t^n}{n!}\). Example5.2(SumofPoissons): Let’snowtrytosynthesizeProperty5.1andProperty5.2.Let\(X\)and\(Y\)bei.i.d.\(Pois(\lambda)\)randomvariables.Considerthedistributionof\(Z\),where\(Z=X+Y\).WeknowbythestoryofaPoissonthat\(X\)and\(Y\)areboth‘countingthenumberofwinninglotteryticketsinalotterywithrateparameter\(\lambda\)’;thatis,wehavemanytrials,withlowprobabilityofsuccessoneachtrial.Whatdoweexpecttobethedistributionof‘combining’thesetwolotteries? Youmayhavesomeintuitionforthisproblem,butalittlebitofworkwithMGFscanhelptosolidifythisintuition(aswellasourunderstandingofMGFs).Specifically,whatwecandoisfindtheMGFof\(Z\),andseeifitmatchestheMGFofaknowndistribution;iftheymatch,byProperty5.1,thentheyhavethesamedistributionandwethusknowthedistributionof\(Z\).WecanfindtheMGFof\(Z\)byProperty5.2:findtheindividualMGFsof\(X\)and\(Y\)andtaketheproduct. So,let’sgettoworkfindingtheMGFof\(X\).WewriteouttheLoTUScalculation,where\(M_x(t)\)denotestheMGFof\(X\): \[M_x(t)=E(e^{tx})=\sum_{k=0}^{\infty}e^{tk}\frac{\lambda^ke^{-\lambda}}{k!}\] Wecantakeoutthe\(e^{-\lambda}\)becauseitdoesn’tchangewiththeindex\(k\). \[e^{-\lambda}\sum_{k=0}^{\infty}e^{tk}\frac{\lambda^k}{k!}\] Wecanthencombinetermsthatareraisedtothepowerof\(k\): \[=e^{-\lambda}\sum_{k=0}^{\infty}\frac{(e^t\lambda)^k}{k!}\] NowrecalltheExponentialseriesthatwementionedearlierinthischapter.Thesumlooksliketheexpansionof\(e^{e^t\lambda}\),sowewrite: \[=e^{-\lambda}e^{e^t\lambda}\] \[=e^{e^t\lambda-\lambda}\] \[=e^{\lambda(e^t-1)}\] ThisistheMGFof\(X\),whichis\(Pois(\lambda)\).Itmaylookalittlestrange,becausewehavethe\(e\)intheexponentofthe\(e\).Weknowthat,since\(Y\)hasthesamedistributionas\(X\),ithasthesameMGF(byProperty5.1),sotheMGFof\(Y\)isalso\(=e^{\lambda(e^t-1)}\)(botharefunctionsofthe‘bookkeeping’variable\(t\)).ByProperty5.2,wetaketheproductofthesetwoMGFs(whichareidentical)tofindtheMGFof\(Z\).Letting\(M_z(t)\)mean‘theMGFof\(Z\)’: \[M_z(t)=e^{\lambda(e^t-1)}e^{\lambda(e^t-1)}\] \[=e^{2\lambda(e^t-1)}\] WhataboutthisMGFisinteresting?NotethatithastheexactsamestructureastheMGFofa\(Pois(\lambda)\)randomvariable(i.e.,theMGFsof\(X\)and\(Y\))exceptithas\(2\lambda\)insteadof\(\lambda\).Theimplicationhereisthat\(Z\simPois(2\lambda)\),andthismustbetruebyProperty5.1becauseithastheMGFofa\(Pois(2\lambda)\)(imaginesubstituting\(2\lambda\)infor\(\lambda\)fortheMGFof\(X\)calculation).Wecanquicklyconfirmthisresultbycomparingthesumoftwo\(Pois(1)\)randomvariablestoa\(Pois(2)\)randomvariableinR. #replicate set.seed(110) sims=1000 #generatether.v.'s X=rpois(sims,1) Y=rpois(sims,1) Z=rpois(sims,2) #seta1x2grid par(mfrow=c(1,2)) #graphics;histogramsshouldmatch(matchthebreak) hist(X+Y,main="X+Y",xlab="", col=rgb(1,0,0,1/4),breaks=0:10) hist(Z,main="Z",xlab="", col=rgb(0,1,0,1/4),breaks=0:10) #re-setgraphics par(mfrow=c(1,1)) Ultimately,wehaveproventhatthesumofPoissonrandomvariablesisPoisson,andevenhaveshownthattheparameterofthisnewPoissonisthesumoftheparametersoftheindividualPoissons.Thisresultprobablymatchesyourintuition:ifweareconsideringthesumofthenumberofsuccessesfromtwo‘rareevents’(i.e.,playingbothlotteries)weessentiallyhaveanother‘rareevent’withthecombinedrateparameters(sinceweare‘in’bothlotteries).Thatis,thestoryofthePoissonispreserved. HopefullythesewerehelpfulexamplesindemonstratingjustwhatMGFsarecapableofdoing:findingthemomentsofadistributioninanewanduniqueway(well,twowaysactually),andlearningmoreaboutspecificdistributionsingeneral.Don’tbescaredbytheuglyintegrationandsummations;usuallythingswillworkoutcleanlyandyouwillgetatractableexpressionforthemomentsofadistributionyou’reinterestedin.MGFsreallyareextremelyuseful;imagineifyouwereaskedtofindthe\(50^{th}\)momentofan\(Expo(\lambda)\).ThatsoundslikeanuglyLoTUSintegral,butsoundsverydoableusingtheMGFapproacheswe’veestablished. Practice Problems 5.1 WehavediscussedhowaBinomialrandomvariablecanbethoughtofasasumofBernoullirandomvariables.UsingMGFs,provethatif\(X\simBin(n,p)\),then\(X\)isthesumof\(n\)independent\(Bern(p)\)randomvariables. Hint:TheBinomialTheoremstatesthat\(\sum_{k=0}^n{n\choosek}x^ny^{n-k}=(x+y)^n\). 5.2 UlysseshasbeenstudyingtheUniformdistributionandhemakesthefollowingclaim.‘Let\(X\simUnif(0,10)\)and\(Y_1,Y_2,...,Y_{10}\)bei.i.d.\(Unif(0,1)\).Thesumofthe\(Y\)randomvariables\(\sum_{k=1}^{10}Y_k\)hasthesamedistributionas\(X\);thisisintuitive,addingup10randomdrawsfrom0to1isthesameasgeneratingonerandomdrawfrom0to10.’TestUlysses’claimusingMGFs. Defendyouranswerto(a.)usingintuition. 5.3 Let\(X\)beadegeneraterandomvariablesuchthat\(X=c\)always,where\(c\)isaconstant.FindtheMGFof\(X\),andusetheMGFtofindtheexpectation,varianceandallmomentsof\(X\).Explainwhytheseresultsmakesense. 5.4 Let\(X\simN(\mu,\sigma^2)\).TheMGFof\(X\)isgivenby\(M_x(t)=e^{\mut+\frac{1}{2}\sigma^2t^2}\)(ingeneral,youcanfindthisandotherusefulfactsaboutdistributionsonWikipedia).Usingthisfact,aswellaspropertiesofMGFs,showthatthesumofindependentNormalrandomvariableshasaNormaldistribution. 5.5 Let\(X\simExpo(\lambda)\)and\(Y=X+c\)forsomeconstant\(c\).Does\(Y\)haveanExponentialdistribution?UsetheMGFof\(Y\)toanswerthisquestion. BHProblems TheproblemsinthissectionaretakenfromBlitzsteinandHwang(2014).Thequestionsarereproducedhere,andtheanalyticalsolutionsarefreelyavailableonline.Here,wewillonlyconsiderempiricalsolutions:answers/approximationstotheseproblemsusingsimulationsinR. BH6.13 Afairdieisrolledtwice,withoutcomes\(X\)forthefirstrolland\(Y\)forthesecondroll.Findthemomentgeneratingfunction\(M_{X+Y}(t)\)of\(X+Y\)(youranswershouldbeafunctionof\(t\)andcancontainun-simplifiedfinitesums). BH6.14 Let\(U_1,U_2,...,U_{60}\)bei.i.d.~\(Unif(0,1)\)and\(X=U_1+U_2+...+U_{60}\).FindtheMGFof\(X\). BH6.21 Let\(X_n\simBin(n,p_n)\)forall\(n\geq1\),where\(np_n\)isaconstant\(\lambda>0\)forall\(n\)(so\(p_n=\lambda/n\)).Let\(X\simPois(\lambda)\).ShowthattheMGFof\(X_n\)convergestotheMGFof\(X\)(thisgivesanotherwaytoseethattheBin(\(n,p\))distributioncanbewell-approximatedbythe\(Pois(\lambda)\)when\(n\)islarge,\(p\)issmall,and\(\lambda=np\)ismoderate). References Blitzstein,J.K.,andJ.Hwang.2014.IntroductiontoProbability.Chapman&Hall/CRCTextsinStatisticalScience.CRCPress.https://books.google.com/books?id=z2POBQAAQBAJ.