IP Fragmentation in Detail - Packet Pushers

The Fragment Offset field (13 bits) is used to indicate the starting position of the data in the fragment in relation to the start of the ... Youarehere:Home/Blogs/IPFragmentationinDetail WhenahostsendsanIPpacketontothenetworkitcannotbelargerthanthemaximumsizesupportedbythatlocalnetwork.Thissizeisdeterminedbythenetwork’sdatalinkandIPMaximumTransmissionUnits(MTUs)whichareusuallythesame.Atypicalcontemporaryoffice,campusordatacentrenetworkprovidedoverEthernetwillhave1500byteMTUs. However,packetsthatareinitiallytransmittedoveranetworksupportingoneMTUmayneedberoutedacrossnetworks(suchasaWANorVPNtunnel)withasmallerMTU.Inthesecases,ifthepacketsizeexceedsthelowerMTUthedatainthepacketmustbefragmented(ifpossible).Thismeansitisbrokenintopiecescarriedwithinnewpackets(fragments)thatareequaltoorsmallerthanthelowerMTU.ThisiscalledFragmentationandthedatainthesefragmentsisthentypicallyreassembledwhentheyreachtheirdestination. Typicallythelanguageusedwhendiscussingfragmentationimpliestheoriginalpacketitselfisfragmented.AsI’vemadeclear,itsthedatawithinitthat’sfragmented–theoriginalpacketisactuallydiscarded. Fragmentationallowsfor; Transportlayerprotocolstobeignorantoftheunderlyingnetworkarchitecture,reducingoverheads.IPAndhigherlayerprotocolstoworkacrossvariableanddiversenetworkpathsandmediumswithouttheneedandoverheadofapathdiscoveryprotocol(butseethePMTUDsection). Fragmentationhasanumberofdrawbackswhichresultinit’susebeingavoidedwherepossible,primarily: ThelossofasinglefragmentresultsinallthefragmentshavingtoberesentwhereareliabletransportlayerprotocolsuchasTCPisinuse(infactthesenderresendsonepacketandfragmentationoccursonceagain).Onlythefirstfragmentcontainsthehighlayerheaderswhichcancauseissueswithfirewalls,middle-boxesandrouters(i.e.NATfunctionality)thatrelyoninspectingthoseheaders.Fragmentationmayresultinoutoforderpacketdeliveryandtheneedforreordering(especiallyifonlysomepacketsarefragmentedoriflinkaggregationorotherpathsplittingtechnologiesareinuse). IPv4 TheIPv4HeaderFieldsUsed TheprocessesoffragmentationandreassemblyinvolveanumberofIPheaderfieldsbeingsetinthefragments.Here’sareminderofallthefieldsandtheirorder,withfragmentationheadershighlighted: Fragmentation’soperationreliesuponthreeIPheaderfields(32bitsintotal),allofwhichwillhaveverydifferentvaluesinthefragmentscomparedtotheoriginalpacket: TheIdentificationfield(16bits)ispopulatedwithanIDnumberuniqueforthecombinationofsource&destinationaddressesandProtocolfieldvalueoftheoriginalpacket,allowingthedestinationtodistinguishbetweenthefragmentsofdifferentpackets(fromthesamesource).ThisdoesnotmeanthesameIDshouldbeusedwhenfragmentingpacketswherethesource,destinationandprotocolarethesamebutthatthesameIDcouldbeusedwhentheyarenot.Tomakethisveryclear,ifthreepacketsaresentfromhostAtohostBandeachmustbefragmentedintofourfragments:thefourfragmentsofthefirstpacketwillsharethesameIdentificationfieldvaluethefourfragmentsofthesecondpacketwillsharethesameIdentificationfieldvalue,whichwillbedifferenttothevalueusedwiththefragmentscreatedfromthefirstpacketthefourfragmentsofthethirdpacketwillsharethesameIdentificationfieldvalue,whichwillbedifferenttothevalueusedwiththefragmentscreatedfromthefirstandsecondpacketsLiketheoriginalpacket,thefirst,reservedbitoftheFlagsfield(3bits)willbe0(unset)andthesecondbit,Don’tFragment(DF),willalsobeunset.Unliketheoriginalpacket,allbutthelastfragmentwillhavethethirdbitofthefield,MoreFragments(MF),setto1.Thelastpacketwillhaveallbitsinthisfieldsetto0justliketheoriginalpacket(unlessitwasafragmentitself).IftheDon’tFragmentflagwassetintheoriginalpacket,thispreventsfragmentationandresultsinpacketsthatrequireitbeingdiscarded.AnICMPerroroftype3:‘DestinationUnreachable’,code4:‘Fragmentationrequired,andDFset’shouldbesenttothesender.SeethefollowingPMTUDsectionformoreonthis.TheFragmentOffsetfield(13bits)isusedtoindicatethestartingpositionofthedatainthefragmentinrelationtothestartofthedataintheoriginalpacket.Thisinformationisusedtoreassemblethedatafromallthefragments(whethertheyarriveinorderornot).Inthefirstfragmenttheoffsetis0asthedatainthispacketstartsinthesameplaceasthedataintheoriginalpacket(atthebeginning).Insubsequentfragments,thevalueistheoffsetofthedatathefragmentcontainsfromthebeginningofthedatainthefirstfragment(offset0),in8byte‘blocks’(akaoctawords).Ifapacketcontaining800bytesofdataissplitintotwoequalfragmentscarrying400bytesofdata,thefragmentoffsetofthefirstfragmentis0,ofthesecondfragment50(400/8).Theoffsetvaluemustbethenumberof8byteblocksofdata,whichmeansthedatainthepriorfragmentmustbeamultipleof8bytes.Thelastfragmentcancarrydatathatisn’tamultipleof8bytesastherewon’tbeafurtherfragmentwithanoffsetthatmustmeetthe8byte‘rule’. Theseheaderfieldvaluesalsochangeeitherbecausetheynormallywouldateveryhop,orasaby-productoffragmentation: TheTotalLengthfield(16bits)changesbasedonthereducedsizeofthedatainafragment(plusIPheader)whichequalsorissmallerthantheMTU.BecausetheFragmentOffsetfieldinthefollowingfragmentsmustbeamultipleof8thefragment’ssizeisn’talwaysaslargeastheMTUallows.NoteinIPv4thisfieldindicatesthetotalpacketsizeincludingtheheader.TheHeaderChecksumfield(16bits)isrecalculatedbasedonthevalueofalloftheotherfieldsintheheaderTheTimeToLivefield(8bits)valueisreducedbyone WorkedExample Enoughofthetheory,letsworkthroughanexampleofa1440bytepacketthatmustberoutedthroughaninterfacewithanMTUof576bytes(thepacketsizeallhostsshouldbepreparedtoaccepttocomplywiththeIPRFC).We’llassumethesmallestpossibleIPheadersizeof20bytesmeaningtheoriginalpacketcontains1420bytesofdata.Thefollowingfragmentsarecreatedwiththesizesandfieldssetasfollows: FirstFragment AnIdentificationfieldpopulatedwithanIDnumberAreservedbitnotset,theDon’tFragmentflagnotsetandaMoreFragmentsflagof1intheFlagsfield,expressedinbinaryas:001TheFragmentOffsetfieldissettozeroasthisisthefirstfragment,expressedinbinaryas:0000000000000ATotalLengthfieldof572expressedinbinaryas:0000001000111100;themaximumpossible576bytepackethasn’tbeenusedbecausethefragmentoffsetfieldinthefollowingfragmentmustbeamultipleof8bytes–astheheaderis20bytes,thispacketcontains552bytesofdata,with868bytesremaining SecondFragment AnIdentificationfieldpopulatedwiththesameIDnumberusedforthefirstfragmentAreservedbitnotset,theDon’tFragmentflagnotsetandaMoreFragmentsflagof1intheFlagsfield,expressedinbinaryas:001TheFragmentOffsetfieldissetto69(552/8),expressedinbinaryas:0000001000101ATotalLengthfieldof572expressedinbinaryas:0000001000111100;thispacketalsocontains552bytesofdata,with316bytesremaining Third&FinalFragment AnIdentificationfieldpopulatedwiththesameIDnumberusedforthefirstandsecondfragmentsAreservedbitnotset,theDon’tFragmentflagnotsetandaMoreFragmentsflagof0intheFlagsfield,expressedinbinaryas:000TheFragmentOffsetfieldissetto138(1104/8),expressedinbinaryas:0000010001010ATotalLengthfieldof336expressedinbinaryas:000101010000;thispacketcontains316bytesofdata Here’sallthatdatainaneasytodigesttable(allvaluesindecimal): FragmentIdentificationFlagsFragmentOffsetTotalLengthOne971210572Two9712169572Three97120138336 Wecanconfirmthesefragmentscontainthe1420bytesofdatafromtheoriginalpacketwithtwodifferentcalculations(thefirstbeingthemostefficient): Addthefragmentoffsetofthefinalfragment,multipliedby8,totheTotalLengthofthefinalfragmentminusitsheaderlength:138x8=1104and336–20=316.1104+316=1420.CalculatethesumoftheTotalLengthofthethreefragmentsandthensubtractthesumoftheheaderlengthofthefragments:572+572+336=1480and20+20+20=60.1480–60=1420. FragmentationwithIPv4isdescribedintheInternetProtocolRFC791:https://tools.ietf.org/html/rfc791. PreventingFragmentation AnodecanpreventpacketsbeingfragmentedbysettingtheDon’tFragment(DF)flaginthosepacketstoavalueof1.PacketsthatmustbefragmentedbuthavetheDFbitsetarediscarded.ThisiscoveredindetailinthefollowingPMTUDsection. PathMTUDiscovery(PMTUD) PMTUDAimstoavoidfragmentationbydynamicallydiscoveringtheMTUofapathbetweentwohostsusingICMPmessages.Thisisdirectlyrelatedtofragmentationandworthcoveringbrieflysoyouhaveafullunderstandingtherelationshipbetweenthetwo. PMTUDOperatesasfollows: ThesendinghostassumestheMTUofthelocalnetworkinterfaceusedtoreachadestinationisvalidalongthepathtothatdestination.IfTCPisusedandthedestinationreportsaMSSthatresultsinalowerMTU,thatisassumedtobevalidalongthepath.ThesendinghostsetstheDon’tFragment(DF)flaginallpacketssenttothedestination.ShouldanyofthepacketsneedtobefragmentedastheytravelalongthepathtothedestinationtheyarediscardedbytherouterinvolvedastheDFbitisset.TheroutershouldthensendanICMPerroroftype3:‘DestinationUnreachable’,code4:‘Fragmentationrequired,andDFset’backtothesource.Thismessageshouldcontaina16bitNext-HopMTUfieldwiththevalue,inbytes,ofthelargestpacketthatcanberoutedtothenexthopwithoutfragmentation(includingIPheader).ItalsoincludestheIPheaderofthepacketthatrequiredfragmentationand64bits(8bytes)ofitspayload,whichwouldnormallycontainthesourceanddestinationportfieldsofthetransportlayerheader.ThisICMPmessageiscalledaDatagramTooBigmessageintheRFC.OnreceivingtheICMPmessage,thesendinghostwouldtypicallyreducethesizeofpacketssenttothedestinationbasedupontheNext-HopMTUfieldvalueinthemessage.ThehostshouldstorethispathMTUinformationinsomewayandthistypically(assuggestedintheRFC)takestheformofaspecificroutingtableentryforthedestinationaddress.Thethetransportlayerprotocol(identifiedusingthediscardedpacket’sIPheaderintheICMPmessage)shouldbeinformedofthereducedpathMTU.Theapplicationsendingthedata(identifiedusingthediscardedpacket’s64bitsofpayloadintheICMPmessage)shouldbeinformedtheoriginalpackethasbeendiscarded.ShouldalowerMTUbeencounteredfurtheralongthepath,thisprocessisrepeated. NotethatPMTUDoperatesindependentlyineachdirectionalongthepathbetweentwohosts(ifbothhostssupportitandhaveitenabled).Thesepaths(andtheMTUofthenetworkstheycompriseof)maybedifferent. Here’showtheDatagramTooBigmessagefieldsarelaidout: PMTUDHelpstoavoidissueswithfragmentationbutofcoursehasissuesofitsown,themostcommonbeingthatICMPmessagesareblockedbyfirewalls,routersorothernetworkdevicessomewhereonthepathbetweentheroutergeneratingthemessage(s)andthehosttheyaredestinedfor.AsthesenderisunawareoftheMTUproblemitretransmitsunacknowledgedpacketsthatthemselvesgounacknowledgedandsoonuntilsomelimitisreachedandtheconnection‘hangs’orstalls.Thisisreferredtoasablack-holeconnection. ThereareanumberofotherconsiderationsandimplicationsofusingPMTUDwhicharewellbeyondthescopeofthisarticlebutifyou’dliketoknowmore,pleaserefertothespecificationinRFC1191:https://tools.ietf.org/html/rfc1191.ICMPv4isspecifiedinRFC792:https://tools.ietf.org/html/rfc792andextendedinRFC4884https://tools.ietf.org/html/rfc4884. PacketizationLayerPathMTUDiscovery(PLPMTUD) ToavoidthePMTUDICMPblockingissue,PLPMTUDusesatransportlayerprotocolthatsupportsacknowledgements(typicallyTCP)toprobethepathtoahost.Insimplisticterms,PLPMTUDoperatesasfollows: multipleprobepacketsofinitiallysmallandlargesizesaresentinsequencethelargestisthesizeoftheMTUofthenetworkinterfaceusedtoreachthedestinationthesmallestis1024bytesprobepacketsthatreachedthedestinationareacknowledgedandthusindicateaMTUsupportedalongtheentirepathunacknowledgedlargeprobesresultinsmallerprobesbeingsentacknowledgedsmallprobesresultinlargerprobesbeingsentthepathMTUissetatavaluebetweenthelargestsuccessfulsmallprobeandlargestsuccessfullargeprobe ThePLPMTUD(whichappliestoIPv4andIPv6)specificationcanbefoundinRFC4821:https://tools.ietf.org/html/rfc4821. IPv6 FragmentationwithIPv6operatesinafundamentallydifferentwaytotothatofIPv4,althoughmostoftheheaderfieldsremainandhavethesamepurpose.IncontrastwithIPv4: Fragmentationcanonlyoccuronthesourcehostmeaningapacketcanonlybefragmentedonceandfragmentationisnotperformedbyroutersorothernetworkdevices.Thusthe‘forward’fragmentationmodelofIPv4,wheretheoriginalpacketisfragmentedwhererequiredandthedataitcontainedispreservedandroutedtowardsthedestination,isreversed.Preventingfragmentationisthussimplyamatterofnotperformingfragmentation.ThereisnoDFflag.ConsequentlyfragmentationcannotoccurunlessPMTUDisusedandnodesmaynotuseanMTUgreaterthantheIPv6minimumof1280bytesunlesstheyimplementPMTUD.PMTUDInIPv6worksinmostlythesamewayasitdoesinIPv4asdetailedearlieralthoughtheICMPerrormessageisdifferentasshowninthefollowingdiagram: Ifapacketistoolargetoberoutedacrossanetwork,theIPv6routerinvolvedmustdiscarditandsendanICMPv6erroroftype2:‘PacketTooBig’,code0tothesender(thecode0meansthereisnocodeandisthereforeignored).TheMTUofthe‘nexthop’link(inbytes)isincludedasisasmuchofthetoobigpacketaspossiblewithouttheICMPv6packetexceedingtheminimumIPv6MTUof1280.ThisrelianceonICMPanda‘backchannel’transmissiontotheoriginatinghosthasanumberofflaws.ThemostsignificantbeingthatICMPmessagesarefrequentlyblockedorratelimitedatvariouspointsinanetworkforperceivedsecuritybenefits.Thismeanstheoriginatinghostwon’treceiveanyPacketTooBigICMPmessagesandwillretransmitthediscardedpacket(s)anumberoftimes,noneofwhichwillarrive.Eventuallytheconnectionwillbeconsideredunusableandclosedandanotherestablishedandtheissuerepeatsendlessly.FragmentationrelatedheaderfieldsareplacedinanextensionheadernamedtheFragmentHeaderwhichisspecifiedwithaNextHeaderfieldvalueof44inthestandardIPv6headerorpriorextensionheader(ifthereisone).Thisheader,likeallextensionheaders,is64bits/8byteslong,inadditiontothestandardheaderwhichresultsinIPv6fragmentationhavingahigheroverheadthanthatofIPv4.YoucouldarguehoweverthatastheIPv4relatedfragmentationfields,totaling32bits,areincludedaspartofthestandardIPv4header,whetherfragmentationisoccurringornot,thatIPv4’ssupportforfragmentationhasahigheroveralloverhead. TheIPv6HeaderFieldsUsed Let’slookatthefieldsoftheIPv6Fragmentextensionheader: TheNextHeaderfield(8bits)isusedtospecifytheheadertypeoftheheaderfollowingthisone.ThiscouldbeanotherIPv6extensionheaderoranupperlayerprotocolheader(thisworksverymuchliketheIPv4header’sProtocolfield).Forexample,ifthisFragmentHeaderisthelastextensionheaderandtheupperlayerprotocolinuseisTCP,thevaluewouldbe6(00000110inbinary).Thefulllistofprotocolnumberassignments,maintainedbyIANAcanbefoundhere;https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml.Thenext8bitsarereserved.TheFragmentOffsetfield(13bits)isusedinthesamewayaswithIPv4.Thenext2bitsarereserved.TheMflagfield(1bit)willbesetto1inallbutthelastfragmentaswiththeMoreFragments(MF)fieldinIPv4.TheIdentificationfield(32bits)issimilartothe16bitIPv4fieldandpopulatedwithanIDnumberuniqueforthecombinationofsourceanddestinationaddresses.TheProtocolfieldisnotconsideredasitiswithIPv4asIPv6doesnothaveone. Theseheaderfieldvaluesalsochangeinthestandardheadercomparedtotheoriginalunfragmentedpacket: ThePayloadLengthfield(16bits)changesbasedonthereducedsizeofthedatainafragmentwhichequalsorissmallerthantheMTU.BecausetheFragmentOffsetfieldinthefollowingfragmentsmustbeamultipleof8thefragment’ssizeisn’talwaysaslargeastheMTUallows.NoteinIPv6thisfieldindicatesthelengthofthepacketminusthestandardheaderbutincludinganyextensionheader(s).TheNextHeaderfield(8bits)valueissetto44toindicatethereisaFragmentHeaderextensionheader.Iftheirarealreadyextensionheaders,theNextHeadervalueinthelastonetocomebeforetheFragmentHeaderwillbesetto44instead. WorkedExample NowletsworkthroughanIPv6exampleusingthesame1440bytepacketasbeforewhichmustberoutedthroughaninterfacewithanMTUof1280bytes(theminimumlinkMTUspecifiedintheRFC).We’llassumethesmallestpossibleIPv6headersizeof40bytesmeaningtheoriginalpacketcontains1400bytesofdata.Thefollowingfragmentsarecreatedwiththesizesandfieldssetasfollows: FirstFragment APayloadLengthfieldvalueof1240–astheFragmentHeaderextensionheaderis8bytes,thispacketcontains1232bytesofdata,with168bytesremainingANextHeaderfieldvalue(inthestandardheader)of44ANextHeaderfieldvalue(intheextensionheader)of6indicatingthetransportlevelprotocolinuseisTCPTheFragmentOffsetfieldissettozeroasthisisthefirstfragmentAMflagof1AnIdentificationfieldpopulatedwithanIDnumber Second&FinalFragment APayloadLengthfieldvalueof176–astheFragmentHeaderextensionheaderis8bytes,thispacketcontains168bytesofdata,ANextHeaderfieldvalue(inthestandardheader)of44ANextHeaderfieldvalue(intheextensionheader)of6TheFragmentOffsetfieldissetto154(1232/8)AMflagof0astherearenomorefragmentsAnIdentificationfieldpopulatedwiththesameIDnumberusedforthefirstfragment Here’sallthatinaneasytodigesttable(allvaluesindecimal): PayloadLengthNextHeaderNextHeaderFragmentOffsetMIdentification12404460197124176446154097124 Wecanconfirmthesefragmentscontainthe1400bytesofdatafromtheoriginalpacketwithtwodifferentcalculations(thefirstbeingthemostefficient): Addthefragmentoffsetofthefinalfragment,multipliedby8,tothePayloadLengthofthefinalfragmentminustheextensionheaderlength:154x8=1232and176–8=168.1232+176=1400.CalculatethesumofthePayloadLengthofthetwofragmentsandthensubtractthesumoftheextensionheaderlengthofthefragments:1240+176=1416and8+8=16.1416–16=1400. FragmentationwithIPv6isdescribedintheInternetProtocol,Version6(IPv6)Specification,RFC8200:https://tools.ietf.org/html/rfc8200#page-15.PathMTUDiscoveryforIPversion6isdetailedinRFC8201:https://tools.ietf.org/html/rfc8201.ICMPv6isspecifiedinRFC4443:https://tools.ietf.org/html/rfc4443andextendedinRFC4884https://tools.ietf.org/html/rfc4884. UDPConsiderations UDPIsoftenusedforreal-timeapplicationssuchasvoiceandvideosofragmentationandreassemblyarehighlyundesirableastheymayintroducedelayandjitterproblemsinadditiontothenumerousotherissuesfragmentationcancause.BeingaconnectionlessprotocolitisunabletousetheMSSmechanismtoevenattempttoavoidfragmentation. ApplicationsrelyingonUDPcanonlydealwiththisattheapplicationlayer.Anumberofprotocolssuchasthereal-timetransportprotocol(RTP)andSessionInitiationProtocol(SIP)canbeusedtoestablishasessionstateandhelpminimiseoravoidtheissuescausedbyIPfragmentation.RTPActuallyhasitsownfragmentationmechanism. Acommonsimplisticapproachistojustuseasmallpacketsize.AllIPv4hosts(includingrouters)shouldbecapableofaccepting576bytepacketsforinstance.TheminimumMTUof1280wouldbeagoodsizetouseinIPv6networks. There’sMore Ifyou’dliketoknowevenmore,readon. tcpdumpandWireshark ThisexamplecapturefromtheWiresharksiteallowsyoutoseewhatfragmentationlookslike‘onthewire’:https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=ipv4frags.pcap.Intheexamplea1400byteICMPpacketmustpassthroughanetworkwitha1000byteMTU. Here’swhatthetwofragmentedpacketslooklikewheninspectedwithtcpdump: 12:03:32.535132IP(tos0x0,ttl64,id46544,offset0,flags[+],protoICMP(1),length996)
  2.1.1.2>2.1.1.1:ICMPechorequest,id5058,seq1,length976
12:03:32.535197IP(tos0x0,ttl64,id46544,offset976,flags[none],protoICMP(1),length452)
  2.1.1.2>2.1.1.1:icmp 1 12:03:32.535132IP(tos0x0,ttl64,id46544,offset0,flags[+],protoICMP(1),length996)
  2.1.1.2>2.1.1.1:ICMPechorequest,id5058,seq1,length976
12:03:32.535197IP(tos0x0,ttl64,id46544,offset976,flags[none],protoICMP(1),length452)
  2.1.1.2>2.1.1.1:icmp You’llnote,aswiththeearlierworkedexample,thefirstfragmentisnot‘stuffed’toitsmaximumpossiblesizeof1000bytesbecausethefragmentoffsetmustbeamultipleof8.TheMoreFragmentsflagisexpressedwitha+forsomereasonandnotethefragmentoffsetisshowninbytes,not8byteblocks.Toexpressthisintableformaswedidearlier: FragmentTotalLengthIdentificationFlagsFragmentOffsetOne9964654410Two452465440122(976bytes/8) Here’showthesamecapturelooksinWireshark,perpacket: Here’showanICMPv4DatagramTooBigresponsetoanoversizedpingisshownattheLinuxcommandline: sudoping-s1460google.co.uk
PINGgoogle.co.uk(216.58.214.3)1460(1488)bytesofdata.
From192.168.1.1(192.168.1.1)icmp_seq=1FragneededandDFset(mtu=1480)
^C
---google.co.ukpingstatistics---
3packetstransmitted,0received,+1errors,100%packetloss,time2012ms 1 sudoping-s1460google.co.uk
PINGgoogle.co.uk(216.58.214.3)1460(1488)bytesofdata.
From192.168.1.1(192.168.1.1)icmp_seq=1FragneededandDFset(mtu=1480)
^C
---google.co.ukpingstatistics---
3packetstransmitted,0received,+1errors,100%packetloss,time2012ms AndwhattheICMPDatagramTooBigmessagelookslikewhencapturedwithtcpdump: sudotcpdump-iwlp3s0-vvnnicmp[icmptype]==3&&icmp[icmpcode]==4
tcpdump:listeningonwlp3s0,link-typeEN10MB(Ethernet),capturesize262144bytes
20:20:09.325974IP(tos0x0,ttl253,id15985,offset0,flags[none],protoICMP(1),length56)
  192.168.1.1>10.11.12.176:ICMP216.58.214.3unreachable-needtofrag(mtu1480),length36
IP(tos0x0,ttl62,id59341,offset0,flags[DF],protoICMP(1),length1488)
  10.11.12.176>216.58.214.3:ICMPechorequest,id11975,seq1,length1468 1 sudotcpdump-iwlp3s0-vvnnicmp[icmptype]==3&&icmp[icmpcode]==4
tcpdump:listeningonwlp3s0,link-typeEN10MB(Ethernet),capturesize262144bytes
20:20:09.325974IP(tos0x0,ttl253,id15985,offset0,flags[none],protoICMP(1),length56)
  192.168.1.1>10.11.12.176:ICMP216.58.214.3unreachable-needtofrag(mtu1480),length36
    IP(tos0x0,ttl62,id59341,offset0,flags[DF],protoICMP(1),length1488)
  10.11.12.176>216.58.214.3:ICMPechorequest,id11975,seq1,length1468 You’llnotethatinformationfromtheIPheaderandthepingisdisplayed. OtherPoints&Considerations Youshouldnotethefollowing: Fragmentation&reassemblyareCPU(andinsomecasesmemory)intensiveandbestavoidedifpossible.Hostsand,lessoften,networkdevicesinvolvedinreassemblymustallocatememorytostoreallthefragmentsuntiltheycanbereassembledandhavenowayofknowinghowmanytheremaybe.Thispresentsmanysecurityandavailabilityrisks.WithIPv6reassemblymaytakenolongerthan60secondsfromthetimethefirstfragmentwasreceived,otherwiseallfragmentsarediscarded.AnICMPerroroftype3:‘TimeExceeded’,code1:Fragmentreassemblytimeexceeded’shouldbesenttothesourceshouldthisoccur.Iffragmentzeroisnotavailable,nomessageissent.Here’swhatthatmessagelookslike: WithIPv4thingsaremorecomplicated;aninitialtimeoutof15secondsissuggestedwhenthefirstfragmentisreceived.TheTTLfieldvalueofeachsubsequentfragmentreceivedisthenusedtoresetthetimervalue,aslongasitishigherthanthecurrentvalue.Thisallowsforagapofupto4.25minutesbetweenthereceiptoffragmentswhichisridiculouslyhigh,particularlyinmodernnetworks(asnotedinRFC4963:https://tools.ietf.org/html/rfc4963).Shouldthetimerexpire,allfragmentsarediscarded.AnICMPerroroftype11:‘TimeExceeded,code1:‘fragmentreassemblytimeexceeded’shouldbesenttothesourceshouldthisoccur.Iffragmentzeroisnotavailable,nomessageissent.Here’swhatthatmessagelookslike: FragmentationaddsbandwidthoverheadsasallthefragmentsrequiretheirownIPheader.TheoriginalpackethaditsownIPheadersotocalculatetheadditionalbandwidthconsumedmultiplythenumberoffragments,minus1(fortheoriginalpacket’sheader)bythesizeofheader.ForIPv6alsoadd8bytesperfragmentfortheFragmentHeaderextensionheaders.Fragmentsaretypicallynotreassembleduntiltheyreachtheirfinaldestination(thereceivinghost).There’snobenefitformostroutervendorsdoingsoconsideringtheperformanceoverhead,possibledelayinreceivingallfragmentsandpossibilityoffragmentlossandretransmission.Reassemblyisnotajobwellsuitedtoadeviceoptimisedtoroutepacketsasquicklyaspossible.Firewallandsecureinspectiondevicevendorsmayreassembletobeabletoinspectthefulloriginaldatapayload.However,inmanycasestheoriginalfragmentsratherthanthereassembledpacketwillberoutedonwardtoavoidthepossibilityandburdensoffragmentationreoccurringatanotherpointonthepathtothedestination.ADCVendorsmayalsoreassembletoremovetheoverheadsfromdownstreamnetworksandserversand/ortoimproveperformance.Iffragmentsarriveoutoforderatafirewallorotherdevicethatinspectslayer4orhigherheadersorinformationtheymaybeblocked,droppedormishandleduntilthefirstfragment(whichcontainsthehigherlayerinformation)arrives.Encryptionanddecryptionand/ortunnellingencapsulationanddecapsulationperformancewillusuallybenegativelyimpactediffragmentationisinvolvedforoneormoreofthefollowingreasons:Fragmentsmayneedtobereassembledbeforetheyareoperatedonpresentingthepossibilityfordelay,reorderingandretransmissionaswellasconsumptionofCPUandmemoryresources.ThereassembledpacketmaythenneedtobefragmentedagainduetoencryptionorencapsulationoverheadreducingtheeffectiveMTU.Evenifreassemblyisnotrequired,furtherfragmentationmayberequiredforthesamereason.Thesoftwarefunctionsusedtohandlethereassemblyand/orfragmentationhavenotbeenoptimisedtothedegreetheprimaryfunctionhas.Different,lessperformanthardwareisusedforthereassemblyand/orfragmentation.Fragmentsthemselvescanbefragmentedagain,inwhichcasetheIdentificationfieldfromtheoriginalfragmentisre-usedwithinthenewfragments.Fragmentationandpoorhandlingoffragmentationhasledtomanysecurityvulnerabilities,seehereforsomeexamples:https://en.wikipedia.org/wiki/IP_fragmentation_attack.AnalternativemethodofreassemblingfragmentscanbefoundinRFC815:https://tools.ietf.org/html/rfc815.SeeRFC4963forinformationonIPv4ReassemblyErrorsatHighDataRates:https://tools.ietf.org/html/rfc4963.SeeRFC4459forinformationonMTUandFragmentationIssueswithIn-the-NetworkTunneling:https://tools.ietf.org/html/rfc4459.ThisFragmentationConsideredHarmfulpaper:http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-87-3.pdfexpandsonsomeoftheissuesfragmentationpresents.SeeRFC6864fortheUpdatedSpecificationoftheIPv4IDField:https://tools.ietf.org/html/rfc6864whichadjuststheIPspecificationsotheIDfieldintheIPheaderisonlyusedforfragmentation.BackuporredundantlinkswithsmallerMTUsthanaprimarylinkaregoingtocausehavoc.Lossyorhighlatencylinksaregoingtohavereally,reallybadperformanceiffragmentsarediscarded.WithTCP,fragmentationorunderlyingMTUissueswon’tusualrevealthemselvesduringconnectionsetupasthepacketsaresmallandrarelycontaindata(useofTCPFastOpen(TFO)beinganexception).Similarly,HTTPrequestsusuallywon’tbeaproblem,thetypicallymuchlargerresponseswillbe.Thiscanmaketroubleshootingdifficult.WithIPv4ICMPmessagesareonlysentabouterrorsinhandlingfragmentzerooffragmenteddatagrams,seetheICMPRFC792:https://tools.ietf.org/html/rfc792formoreonthis.PLPMTUDProbepacketlossdoesnothaveanimpactonthecongestionwindow.ECMPLayerfourloadbalancingtowardsserversoftencausesissueswithfragmentationasthefirstfragmentcontainingthetransportlayerprotocolheaderscanbeloadbalancedtoadifferentservertotheothers,whichdon’t.ECMPMayalsodosomethingsimilarwiththeICMPmessagesrelieduponbyPMTUDandloadbalancemessagestothewrongserverastheECMPalgorithmwillfallbacktolayerthreehashingratherthaninspectingtheICMPpayloadandusingtheportinformationtheirtoperformlayerfourhashing.YoucantestPMTUDandfragmentationsupportacrossapathfromyourhosttothisone:http://icmpcheck.popcount.org. AboutStevenIvesonThelastoffourchildrenoftheseventies,StevewasborninLondonandhasneverbeentoofarfromashooting,bombingorriot.He'snowgratefultoliveinasmalltowninEastYorkshireinthenortheastofEngland. He'sworkedintheITindustryforover25yearsinavarietyofroles,predominantlyindatacentreenvironments.Morerecentlyhe'swidenedhisskillsettoembraceDevOps,Linux,containers,automation,orchestration,cloudandmore.He'sbeenawardedF5NetworksDevCentralMVPstatusseventimes,in2014,2016,2017,2018,2019,2020and2021.DetailsofhisF5relatedbookscanbefoundhere.YoucanfindhimonTwitter:@sjiveson. Comments Amazingarticle.Helpedmetotrulyunderstandtheprocessoffragmentation,whichisnotclearlyportrayedanywhereelse! Thankyou,gladIcouldhelp. EmailFacebookLinkedInRSSTwitterYouTube HeavyNetworking HeavyNetworking621:GetScalableNetworkAutomationWithItential(Sponsored)March11,2022 YouTube UnderstandingDataCenterFabrics07:LinkStateProtocolInTheUnderlayMarch17,2022 DayTwoCloud DayTwoCloud138:RethinkingLogsAndAnalysisWithvRealizeLogInsightCloud(Sponsored)March16,2022 NetworkBreak NetworkBreak373:GoogleBuysMandiantToBolsterSecurityOps;NetworkOSDENT2.0ReleasedMarch14,2022 BriefingsInBrief TechBytes:IntegratingDigitalExperienceManagementAndCloud-DeliveredSecurity(Sponsored)March14,2022 IPv6Buzz IPv6Buzz096:GreenfieldVs.BrownfieldIPv6-OnlyDeploymentsMarch10,2022 FullStackJourney FullStackJourney064:ShouldYouEmbraceChaosEngineering?March15,2022 TheCommunityShow NetworkNeighborhood04:WeTheSalesEngineersWithRamziMarjabaNovember21,2019 RecentComments GoranMekićonIPv6Buzz093:DissectingDHCPv6Yellowfin34onUbuntu:ExtendyourdefaultLVMspaceNinadonUbuntu:ExtendyourdefaultLVMspaceguest1onUbuntu:ExtendyourdefaultLVMspaceGEORGEWENZELonHeavyNetworking608:EverythingYouEverWantedToKnowAboutNAC(AndThenSome)TonyonUbuntu:ExtendyourdefaultLVMspace PacketPushersPodcast PacketPushersArticles Search WebsiteInformation Connect FacebookLinkedInRSSTwitterYouTube