Official Guidance：Understanding Surge

Chapter 4 Forwarding, proxy and rule systems ... The core function provided by TCP layer is reliable transmission, and upper layer ... OfficialGuidance：UnderstandingSurge 1Overview 2Takeover 2.1Proxyservicetakeover(Method1) 2.1.1Whatisaproxyprotocol? 2.1.2HTTPProxyandTCPProxy 2.1.3Excludesimplehostnamesandbypassproxysettingsforthesehosts 2.2VirtualNICtakeover(Method2) 2.2.1FakeIP 2.2.2tun-excluded-routesandtun-included-routesoptions 3Processing 3.1Distinguishingrequests 3.1.1HTTP 3.1.2HTTPS 3.2ModifyingRequestsandResponses 3.2.1Redirects(URLRewrite,MapRemote) 3.2.2MapLocal(APIMock) 3.2.3HeaderRewrite 3.2.4JavaScriptScriptModification 4Forwarding,proxyandrulesystems 4.1RuleSystem 4.2Policy 4.2.1Built-instrategy 4.2.2ProxyPolicy 4.2.3PolicyGroup 4.3Rules 4.3.1DomainNameRules 4.3.2IPaddressrule 4.3.3HTTPRelatedRules 4.3.4OtherRules 4.3.5Ruleset 4.3.6Logicrules 5TLS,HTTPSandMITM 5.1TheRelationshipbetweenTLSandHTTPS 5.2TheroleofTLS 5.2.1DataEncryptionandIntegrityProtection 5.2.2Confirmingtheidentityofthetargethost 5.3MITMattack 5.3.1Surge’sMITMprocess 5.3.2Publicrootcertificate 5.3.3MITMAttackCountermeasures 5.4AdditionaldetailsofTLS 5.4.1CommonHTTPSErrors 5.4.2SNI 5.4.3ForwardSecrecy 5.4.4TLSCipherSuite 6DNS 6.1QueryInParallel 6.2OptimisticDNS 6.3LocalMapping 6.4Usingsystem’sresolution SurgeWebsite OfficialGuidance：UnderstandingSurge OfficialGuidance：UnderstandingSurge SurgeNetworksInc. 2020-08-27 Chapter1Overview SurgeisanetworkingtooloniOSandmacOSplatformswithfourcorecapabilities. Takeover:Youcantakeoverthenetworkconnectionsentbythedevice.SurgesupportsbothproxyserviceandvirtualNICtakeover. Processing:Youcanmodifythenetworkrequestsandresponsesthathavebeentakenover.ThisincludesURLredirection,localfilemapping,custommodificationusingJavaScript,andmanyothermethods. Forwarding:Youcanforwardthetakenovernetworkrequeststootherproxyservers.Thiscanbeglobalforwardingorwithaflexiblerulesystemtodetermineanoutboundpolicy. Intercept:Youcaninterceptandsavespecificdataofnetworkrequestsandresponses,andyoucanalsodecryptHTTPStrafficwithMITM. ThesefourcapabilitiesformthecoreworkflowofSurge.ButSurge’sskillsarenotlimitedtothesefourpoints.Forexample,youcancustomizeDNSservers,configureDNS-over-HTTPSglobally,andsoon. Chapter2Takeover InorderforSurgetoperformsubsequentfunctionssuchasforwarding,modifying,andintercepting,itisfirstnecessaryforSurgetotakeoverthenetworkconnection. UndermacOSandiOS,therearethreewaystohaveanetworkconnectionissuedbyaprogramtakenoverbyanotherprogram,ratherthansendingdatadirectlytoaphysicalnetworkcard. Proxy Ifthesystemisconfiguredwithaproxyserver,theprogramwillnotdirectlyconnecttothetargetserverwhenexecutinganetworkrequest,butwillinsteadgenerateaconnectiontotheproxyserver.Usingthisfeature,itispossibletotakeovernetworkrequestsbystartingalocalproxyserviceandconfiguringthesystemproxyon127.0.0.1. However,thisapproachrequirestheprogramitselftosupporttheproxymechanism.Thesystem’sproxysettingssimplyinformtheprogramthatitshouldusetheproxy,requiringtheprogramtocompletethesubsequentlogicoftheproxyitself.Thegoodnewsisthatforthevastmajorityofapplicationswithauserinterface,thedeveloperdoesnotneedtodoanyadditionalworktosupportproxiesbecausethesystem’shigh-levelnetworkingframework(Cocoa/CocoaTouch)isusedduringdevelopment. Asforcommand-lineapplications,sincetheyusethePOSIXinterfacetomakenetworkrequests,theinterfacedoesnotprovideembeddedsupportforproxy,sothedevelopershavetosupporttheproxyserverbythemselves,whichleadstoinconsistencyinthesupportfortheproxyandthespecificbehaviorofvariouscommand-lineapplications.Also,sincemostcommand-lineprogramsdonothavespecialhandlingformacOS,theyignoretheproxyserversettingsinthesystemconfiguration.Mostcommand-lineprogramsneedtoconfiguretheproxythroughtheenvironmentvariableshttps_proxyandhttp_proxy,whileothersneedtosetitbymodifyingtheconfigurationfile. Therearealsoasmallnumberofapplicationsthatcannotbetakenoverinthiswayduetoacompletelackofproxyserversupport. VirtualNetworkInterface(VIF) AlmostallmainstreamoperatingsystemshaveTUNandTAPvirtualnetworkinterfaces,whichwereoriginallydesignedtoprovideVPNsupport.BycreatingavirtualNIConthesystemandconfiguringtheglobalroutingtable,allnetworkrequestscanbetakenover. Thismethodisnotvisibletotheapplication,sothereisnoneedfortheapplicationtosupportitactively,andalmostanyapplicationcanbetakenoverinthisway.UnlesstheapplicationexplicitlyspecifiesaphysicalNIC,bypassingthedefaultvirtualNIC. SocketFilter ThisisakernelfeatureofmacOSthatcantakeoverrequestsbyinjectingaKernelExtension(kext)tohookallsocketcalls. Thiscanbeusedtoforceallnetworkrequeststobetakenoverbyallprogramsonthesystem,exceptforsomeofthesystemitself.ProxifierandLittleSnitch,forexample,usethismethodtotakeoverthenetwork. Thereareadvantagesanddisadvantagestoeachofthethreemethods. Method1hasthebestperformanceandistheleastintrusivetothesystem,butisnotsupportedbysomeapplications.2. Method2hasslightlylowerperformancebecausetheinterceptedtrafficisIP-layerpacketsthatrequireaTCPstacktobereassembled,resultinginadditionalperformanceoverhead. Method2isthemostintrusivetothesystem.TheKernelExtensionmaycausesystem-wideinstability,andApplehasconfirmedthatthesupportforSocketFilterwillberemovedinfuturemacOS. Surgemainlyusesmethod1totakeovernetworkrequests.Method2supplementsthisbytakingoverapplicationsthatdonotsupportproxies. ForSurgeiOS,whenstarted,itregistersitselfasaproxyserver,andaTUNvirtualnetworkcardisestablishedusingtheNetworkExtensionAPI. ForSurgeMac,enablingthe“SetasSystemAgent”optionwillregisteritselfasaproxyserver(Method1),andenablingthe“EnhancedMode”optionwillcreateavirtualnetworkcard(Method2). TheabovedescriptionisfortheSurgetakeoverofalocalprogram.WhenusingSurgetotakeoveranetworkrequestfromanotherdevice. DuetothesystemlimitationsofiOS,youcanonlyrelyonusingmethod1asaproxyservertotakeoverrequestsfromanotherdevice.(Modifythetargetdevice’sproxyserversettings) SurgeMacinadditiontousingmethod1,youcanalsorelyonmethod2totakeoverarequestfromanotherdevice.(Modifythedefaultroutesettingsforthetargetdevice) 2.1Proxyservicetakeover(Method1) 2.1.1Whatisaproxyprotocol? Aproxyisamechanismthathasbeenaroundsincethedawnofcomputernetworks.Aproxyserverisamessengerthatrelaysdatabetweenanapplicationandatargetserver. Whenusingaproxyserver,someadditionalworkisrequiredinadditiontosendingtherawdata. Informtheproxyserver,thehostnameandportnumberofthetargetserver. Sendauthenticationinformationforproxyserverauthentication.(optional) Encryptionofdatatransmissions.(optional) TheonlytwoproxyprotocolswithRFCspecificationsareHTTPproxyandSOCKSproxy.SOCKSprotocolhas3versions:v4,v4aandv5.(SOCKS5isusedbymacOS) InadditiontoproxyprotocolswithRFCspecifications,therearemanycustomproxyprotocols,suchasshadowsocks,Snell,etc.However,sincesystemsandapplicationsdonothavebuilt-insupportfortheseprotocols,theyneedtobeconvertedtostandardHTTPorSOCKS5proxyservicesbyaclientapplicationforusebyapplicationsandsystems.Surgecanalsoactassuchaconverter,providingproxyservicestosystemsandapplicationswithHTTPproxyandSOCKS5proxyprotocols. TheHTTPSandSOCKS-TLSproxiesdonothaveanRFCspecificationbutareencryptedwithaTLSlayerovertheoriginalprotocol. 2.1.2HTTPProxyandTCPProxy TheHTTPproxycanonlyforwardHTTPprotocolrequests(unlesstheHTTPproxyadditionallysupportstheCONNECTmethod).WhenanHTTPproxyisused,afullHTTPrequestissenttotheHTTPproxy,theproxyserverreceivestherequest,forwardsit,getstheHTTPresponse,andforwardsittotheclient.Therefore,theHTTPproxyissession-based,andasingleHTTPproxyconnectioncankeepforwardingdifferentHTTPrequests,andtheseHTTPrequestsmaynotevenbetothesametargethost. AllotherproxyprotocolsareTCPdataflowproxies,whichsimplyrelayTCPdataflow,whichmeansthattheproxydoesn’tcareaboutorunderstandexactlywhatisbeingdelivered. NotethatthetraditionalHTTPproxyisnotavailablewhenweareconnectingtothetargetserverusingHTTPSprotocol,andwedonotwanttheproxytobeawareofwhatisbeingforwarded,sotheplaintextofwhatisbeingpassedisnotvisibletotheproxy.Forthisreason,theHTTPproxyprotocoladdstheCONNECTmethod,whichturnsanHTTPproxyintoaTCPdataflowproxyforhandlingHTTPSrequests.Sonow,theHTTPproxycanalsobeusedtoforwardanyTCPprotocol. 2.1.3Excludesimplehostnamesandbypassproxysettingsforthesehosts InthemacOSnetworksettings,thereareoptionsto“Excludesimplehostnames”and“Bypassproxysettingsforthesehostsanddomains”,which,liketheproxysettingsitself,only“inform”theapplicationthatitshouldbehaveinthisway. Aswiththeproxyimplementation,mostapplicationswithauserinterfaceautomaticallysupportthesetwooptionsbecausetheyaredevelopedusingthesystem’shigh-levelwebframework(Cocoa/CocoaTouch).Almostallcommand-lineapplicationsdonotsupportthesetwosettings. Theexclude-simple-hostnamesandskip-proxysettingsinSurge’s[General]configurationcorrespondtothesetwooptions,whichwillbeappliedtothesystemwhen“Setassystemproxy”ischecked. Notethatifbothmethod1andmethod2arestartedatthesametime,networkconnectionsexcludedbythesetwosettingswillalsobetakenoverbymethod2.(OnSurgeMacs,itdoesworkifEnhancedModeisnotturnedon) 2.2VirtualNICtakeover(Method2) 2.2.1FakeIP UnderthePOSIXspecification,makingnetworkrequestsrequireDNSresolutionthroughgethostbynameandsimilarmethods,andthenconnecttotheacquiredIPaddress,causingaproblemwhenusingMethod2totakeovertherequest:DNSresolutionmustbeperformedfirst. However,ifthenetworkrequestSurgedecidestopassittoaproxyserverforforwarding,alocalDNSlookupismeaningless,andinsomecases,thedomainnamemaynotberesolvedlocallyatall. Tosolvethisproblem,whenSurge’sVIFreceivesaDNSquery,itdoesnotperformarealDNSquerybutreturnsafakeIPaddress(usually198.18.x.x,theaddressblockwhichisnotusedonthepublicnetwork).WhenaTCPorUDPpacketsenttothefakeIPisreceivedlater,thefakeIPistranslatedbacktotheoriginaldomainnameforfurtherprocessing. TheTTL(Timetolive,whichcanberoughlyinterpretedasexpirydate)oftheDNSresponsereturnedbySurgeisonly1second,sotheresultisthrownassoonasyouuseit,youdon’thavetoworryaboutnetworkanomaliesafterSurgeshutdownduetofakeIP.(However,ithasbeenobservedthatsomesmarthardwaredoesnotfollowTTLcorrectlyandre-doDNSlookup,whichcanbesolvedbyrebootingthedeviceusually.) InearlierversionsofSurge,itwasprudenttoonlyreturnfakeIPsforhostnamesmarkedasforce-remote-dnsintherules,butsincethisoptionwasoftenanuisancetousers,ithasnowbeenremoved,andfakeIPaddressesarereturnedforallhostnames.Thealways-real-ipoptioninConfigure[General]isusedtooverridethisbehavior,andSurgedoesnotreturnfakeIPforhostnamesthatappearinthisoption,insteadforwardstheDNSquerytoaDNSserverforarealIPaddress. 2.2.2tun-excluded-routesandtun-included-routesoptions WhencreatingavirtualNIC,Surgewilladdadditionalroutingtablesbasedonthesetwooptions,tun-excluded-routesiseasiertounderstand,someusersmayaskwhythereisatun-included-routesoption,shouldn’tallroutesbeincludedbydefault? Formainstreamoperatingsystems,thepriorityofroutingtableentriesisdeterminedbythesubnetcoverageoftheentry.Thesmallerthecoverage,thehigherthepriorityoftheroutingtableentry,notinasequentialortop-downorder. SoevenifSurge’sVIFisconfiguredwithadefaultroutingtableentryof0.0.0.0/0,thephysicalNICitselfhasaroutingtableentryforthecurrentsubnet(e.g.,192.168.1.0/24),whichhasahigherprioritywithasmallcoverage.Soallnetworkconnectionssentto192.168.1.xwillstillnotbepickedupbySurge.Iftun-included-routes=192.168.1.100/32isconfigured,thenthisroutingtableentryhasthehighestpriorityforthesmallestcoverage,allowingnetworkconnectionstothatIPtobepickedupbySurgeaswell. Chapter3Processing Beforetheconnectionscanbeprocessed,thenetworkconnectionsfirstneedtobeclassified,andtherearefourmaintypesinthecurrentversionofSurge. HTTPConnection:AconnectiontakenoverbytheSurgeHTTPproxyservice.TheDashboarddisplaysthefullURLandHTTPmethodnameoftherequest(GET,POST,HEAD,PUT,DELETE,TRACE,OPTIONS,etc.) HTTPSconnection:AconnectionthatistakenoverbytheSurgeHTTPproxyserviceandisconvertedtoaTCPstreamusingtheCONNECTmethod.ThetargethostnameandportnumberoftherequestaredisplayedintheDashboard,andthemethodisshownasHTTPS. TCPconnection:ATCPconnectiontakenoverbySurgeVIForSurgeSOCKS5proxyservice.IntheDashboard,thetargethostnameandportnumberoftherequestaredisplayed,andthemethodsaredisplayedasTCPandSOCKS. UDPSessions:UDPpacketshostedbytheSurgeVIFandsenttothesameaddressandportnumberformaUDPsession. ForType1connections,requestscanbemodified,forwarded,andfetchedusingSurge’scapabilities.ForType2connections,afterdecryptionusingMITM,theyareexposedasastandardHTTPconnections.Fullcapabilitiesarealsoavailable.ForType3connections,generally,onlyforwardingispossible. 3.1Distinguishingrequests 3.1.1HTTP Type3connectionsmentionedabovemayalsobeanHTTPconnection,butsinceSurgedoesnothaveawaytoaccuratelyidentifyit,itdoesnotactivelyusetheHTTPenginetoprocessit,ortheconnectionmaybebrokenduetoinabilitytoprocessit. Someusershavesuggestedthatpacketidentificationcanbedonetoautomaticallydeterminethespecificprotocol,butSurgeisnotdesignedtodothisfortworeasons. AlthoughmostTCP-basedprotocolsthefirstpacketissentfromtheclientside,thisisnotrequiredbytheTCPstandard.Forafewprotocols,suchasTelent,thefirstpacketissentfromtheserverside. IfSurgewantstodeterminewhethertheconnectionisHTTPbyidentifyingthepacket,itmustwaitfortheclienttosendthefirstpacket,whichwillresultintheclientwaitingiftheservershouldsendthepacketfirst.(Thiscanbehandledbyaddingawaittimeout,butit’sinelegant.) TherearecustomprotocolsthatsendoutpacketswithHTTPrequestheaders,butthesubsequentdatadoesnotfollowtheHTTPspecification,whichisperfectlyacceptablebyTCPstandardsbutwillcauseanexceptionifitishandledbytheSurgeHTTPengine. So,Surgegivesthechoicetotheuserbyprovidingtheforce-http-engine-hostsparameter,whichforcestheHTTPenginetoprocessanyhostnamethatappearsintheparameter,evenifitisaType3connectiontakenoverbySurgeVIForSurgeSOCKS5.AllHTTPadvancedfeaturescanbeenabled.However,iftheconnectionisnotaconnectionusingtheHTTPprotocol,itwillcausetheconnectionbroken. Inaddition,ifSurgefindsanHTTPrequestheaderorresponseheaderinaTCPconnection,itmakesanoteof“HTTPrequestheaderfoundintherawTCPconnection.”and“HTTPresponseheaderfoundintherawTCPconnection.”,andextractstherequestandresponseheadersforsimpleanalysis.PleasenotethatthisfunctiononlyanalyzesthefirstpacketoftheTCPstreaminbothdirections,andsincetheHTTPprotocolusuallymultiplexesTCPconnections,subsequentHTTPrequestsarenotvisibletothisfunction. 3.1.2HTTPS AconnectiontakenoverbytheHTTPproxyserviceusingtheCONNECTmethodisdeterminedbySurgetobeanHTTPSconnection,butitmayalsobeaTCPconnectionotherthanHTTPS. Therefore,bydefault,SurgeonlysimplyforwardsTCPstreamsforsuchconnections.Ifthetargethostnameisinthehostnameconfigurationof[MITM],SurgewillperformHTTPSdecryption. 3.1.2.1MITMDecryptionforTCPConnections Whenthetcp-connectionoptioninthe[MITM]segmentoftheconfigurationisturnedon,ifthehostnameofaType3connectionappearsinthehostnameconfigurationof[MITM],SurgealsoperformsHTTPSdecryptionandpassesittotheHTTPengineforprocessing,enablingallHTTPadvancedfeatures. However,iftheconnectionisnotHTTPS,itwillcausetheconnectionbroken. 3.2ModifyingRequestsandResponses AllofSurge’smodifyingcapabilitiesaredesignedfortheHTTPprotocol.UnencryptedHTTPSconnectionsandTCPconnectionswillskipthisstep. ThecurrentmodifyingcapabilitiesofferedbySurgeinclude: Redirection(URLRewrite,MapRemote) Localdocumentmapping(MapLocal,APIMock) RequestHeaderandResponseHeaderRevision(HeaderRewrite) JavaScriptscriptmodification Amongthem,scriptmodificationisthemostpowerful,andallothercapabilitiescanbeachievedindirectlythroughscripts.However,sincescriptingistediousandrequiresabitmorecostatruntime,simplerequirementsshouldbeimplementedinotherways. 3.2.1Redirects(URLRewrite,MapRemote) SurgeprovidestwoimplementationsofHTTPredirection. Requestheadermodification:Thisisachievedbydirectlymodifyingthecontentsoftherequestheader.Theclientprogramisnotawareofthisredirect.Inordertoensurecorrectbehavioraftertheredirect,SurgeautomaticallyoverwritestheHostfieldoftherequestheaderwiththehostnameportionoftheURLaftertheURLismodified.Thisbehaviorisnotperformedwhentheredirectisperformedbyascript. Return302,307Response:Returnsa302/307HTTPresponsedirectly.TheclientprgoramneedstosupportHTTPredirection. 3.2.2MapLocal(APIMock) Dependingonthepurpose,thisfunctionisknownbydifferentnamesindifferentsoftware,suchasMapLocal,APIMock,etc.However,itactuallyreferstothesametypeoffunction:returnapredefinedresponseinsteadofmakingarealHTTPrequest. SurgeautomaticallyselectstheappropriateContent-TypefortheHTTPresponseheaderfieldbasedonthelocalfileextension,oryoucancustomizetheresponseheaderfieldtooverridethisbehavior. 3.2.3HeaderRewrite Thisfunctionisusedtomodifythefieldsintherequestheader,andsupportstheadd,del,andreplaceoperations. Whenanaddoperationisused,ifthefieldnamealreadyexists,afieldwiththesamenameisappended,whichisallowedbytheHTTPstandard.Theservershouldbeabletounderstanditbysplicingmultiplefieldswiththesamename.However,sincesomeHTTPserversdonotfollowthisspecificationproperly,itisgenerallyrecommendedtouseacombinationofdelandaddunlessthereisaspecificneedtodoso. 3.2.4JavaScriptScriptModification Scriptmodificationprovidesthemostcomprehensivemodificationcapability.Youcanmodifytherequestandresponseheaderfieldsanddatabodiesatwill,butpleasenotethatcurrently,SurgeonlysupportsscriptreadingandmodificationofUTF-8encodeddatabodies. Pleaserefertothemanualfordetailedinstructionsonhowtousethescript. Chapter4Forwarding,proxyandrulesystems Therequestwillbeforwardedafteritisprocessed.IfSurge’soutboundmodeissettoDirectOutbound,therequestwillbesentdirectlytothetargetserver,andiftheoutboundmodeissettoGlobalProxy,itwillbeforwardedtotheproxyserver. WhentheOutboundModesettingissettoRule-basedProxy,theforwardingpolicywillbedeterminedbasedontheconfiguredrules. 4.1RuleSystem Therearetwobasicconceptsintherulessystem:policyandrule. Policy:describesthewayinwhichSurgecarriesoutitsredirection,withthreecategories. Built-inpolicies:DIRECT,REJECT,RECJCT-TINYGIF,REJECT-DROP. Proxypolicies:eachpolicycorrespondstoaproxyservice Policygroup:selectafinalpolicyfromthesub-policiesaccordingtocertainrules. selectpolicygroup:selectthepolicythroughtheUImenu. url-testpolicygroup:Selectthepolicywiththelowestlatency. fallbackpolicygroup:Selectthepolicywiththehighestpriorityamongtheavailablepolicies. ssidpolicygroup:SelectthepolicybasedonthecurrentWi-FiSSID. load-balancepolicygroup:randomlyusesasub-policywithoptionalavailabilitychecking. Rule:aruleconsistsoffourparts:type,condition,policy,andparameter.Whentheconditionismet,therulematches,usingthepolicyspecifiedbytherule. 4.2Policy Theuseofthevariouspoliciesisspecifiedbelow. 4.2.1Built-instrategy Thebuilt-inpoliciesareprovidedbySurgeanddonotvarywithconfiguration. DIRECT:Sendtherequestdirectlytothetargetserver REJECT:RejecttherequestandreturnanerrorpagewhentheconnectiontypeisHTTP.(Thisbehaviorcanbecontrolledbytheshow-error-page-for-rejectparameter) REJECT-TINYGIF:Rejecttherequestandreturna1pxGIFimageresponsewhentheconnectiontypeisHTTP.Iftheconnectionisofanothertype,itissimplydisconnected.Thispolicyismainlyusedforwebadblocking. REJECT-DROP:Rejecttherequest.UnlikeREJECT,thispolicywillsilentlydiscardtheconnection.Becausesomeapplicationshaveveryviolentretrylogic,theywillimmediatelyretryafterafailedconnection,resultinginastormofrequests. SincetheoperatingsystemdoesnotprovideabandonmentAPIforuser-spaceprogramsockets,Surgesilentabandonmentisimplementedbyleavingthesocketidleforaperiodoftimebeforeclosingit. Also,ifalargenumberofrequeststoahostnametriggertheREJECT/REJECT-TINYGIFpolicywithinashortperiodoftime(thethresholdis10timeswithin30secondsinthecurrentversion),SurgewillautomaticallyupgradethepolicytoREJECT-DROPinordertoavoidwastingalotofresources. 4.2.2ProxyPolicy Proxypoliciesaredefinedbytheuser,andeachpolicydescribesaproxyservice.Requestsareforwardedwhenaproxypolicyisused. Asimpleproxypolicydefinitionlineisasfollows. ProxyA=http,11.22.33.44,8080,username=user,password=pass ProxyAisthepolicynameforusebytherulesandpolicygroups.Thefirstparameteristhetypeofproxyprotocol,currently,Surgesupportsthefollowingtypesofproxyprotocols:http,https,socks5,socks5-tls,ss,snell,vmess,trojan,andtwospecialtypesexternalanddirect. Thesecondparameteristheproxyserverhostname,thethirdistheproxyserverportnumber,followedbythekey=valueparametertable,whichisprovideddifferentlydependingontheprotocoltype. Differenttypesofproxyprotocolshavetheirownspecificparameters,andsomeparametersarecommontoallproxypolicies. tfo:EnablingTCPFastOpen,TFOallowstheTCPhandshaketopassthefirstpacketduringhankshake,thusreducingthetimeoverheadoftheproxyprotocolhandshake.However,sincethisisanewfeatureintroducedin2014,ifanyofthedevices(routes/firewalls)betweenthetargetserverdoesnotsupportthisfeature,itwillcauseanexception.IthasbeenobservedthatmostISP’snetworkshaveachanceofencounteringanerror,soitisnotrecommendedtoenableitunlessitisaLANproxyserver. underlying-proxy:Useaproxythroughanotherproxy,whichmaybereferredtoasaproxychaininothersoftware.(CurrentlyonlyavailableinSurgeiOSbeta) interface:ForcetheproxytouseaspecificNIC. allow-other-interface:IfitisfalseandthespecifiedNICdoesnotexist,theconnectionwillbebroken. Ifalways-use-connect=true,theremoteHTTPproxywillbeusedasastandardTCPproxy. 4.2.2.1directtype(SurgeMaconly) Thisisaspecialtype,notstrictlyaproxy,usedtoforceaspecifiedNICtobeusedforrequests. PolicyName=direct,interface=en2,allow-other-interface=false NotethatduetorestrictionsintheDarwinkernel,theselectedNICmusthavearoutingtableentryforthedestinationaddress,otherwisetheNICcannotbeused. Forexample,ifyouareconnectedtobothawiredandwirelessnetwork,wecanseewiththenetstat-rncommandthatbothcards,en0anden1,havedefaultfull-coverageroutingtableentries,butwithdifferentpriorities. Internet: DestinationGatewayFlagsNetifExpire default192.168.20.1UGScen0 default192.168.20.1UGScIen1 Inthiscase,youcanusethedirectpolicytofreelyselecten0oren1. UsedwithVPN AverysmallcoverageroutingtableisaddedaftersomeVPNconnectionestablishedandcanbeusedtoaccessspecificintranetresourcesduetothesmallcoveragepriority.Whenenhancedmodeisnotenabled,iftheegresspolicyisDIRECT,Surgewillfollowthisbehaviorforlocalforwardingaswell. However,ifenhancedmodeisenabled,sinceSurge’sVIFhasbeenconfiguredasthedefaultroute,theegressconnectionwillbeforcedtobindtotheoriginaldefaultcard,ignoringtheroutingtable,inordertooriginatetheconnectionfromthelocalphysicalNIC.ThiswillcausetheVPN’sintranetroutingtablefailingtowork,whichcanbesolvedbyforcingtheVPN’sutundevicetobindusingthedirecttypepolicy. [Proxy] CorpVPN=direct,interface=utun1,allow-other-interface=true [Rule] DOMAIN-SUFFIX,internal.corp.com,CorpVPN 4.2.2.2externaltype(SurgeMaconly) TheexternaltypepolicyallowsSurgetoworkmoreeasilywithotherproxyprocotolclients. Thisfeatureiscurrentlyonlyavailablebydirectlyeditingtheprofile. External=external,exec=“/usr/local/bin/local”,args=“-c”,args=“/usr/local/etc/config.json”,local-port=1080,addresses=11.22.33.44 Theargsandaddressesfieldscanbeappendedrepeatedly. WhenthispolicyisusedSurgedoesthefollowing. 1.Starttheexternalprocesswiththeexecandargsparameters,andthenforwardtherequesttoSOCKS5127.0.0.1:[local-port]. 2.Iftheexternalprocessisterminated,itwillautomaticallyrestartwhenthepolicyisusedagain. 3.SurgeautomaticallyexcludestheaddressesintheaddressesparameterfromtheVIFroutingtablewhenEnhancedModeisstarted.(EntertheIPaddressoftheproxyserverbeingusedinthisfield.) 4.TheDIRECTpolicyisalwaysusedwhenrequestsfromexternalprocessesstartedbySurgeareprocessedbySurgeVIF.(Inordertocopewithpluginrequestslikeobfs-local,childrenoftheexternalprocessarealsohandledinthesameway) 5.Surgeautomaticallyshutsdownallexternalprocesseswhenitexits,andEnhancedModeautomaticallyclearstheroutingtableofincomingrouteswhenitisturnedoff. Thefunctionsof3and4aboveoverlap,pleasetrytousetheaddressesdeclarationtoexcludeaddressesusedforTUNprocessingtoreducesystemoverhead,thefunctionof4isanadditionallayerofprotection. 4.2.3PolicyGroup Surgeoffersmanydifferenttypesofpolicygroupstomeetthedifferentneedsofvariousscenarios. 4.2.3.1ConnectivityTesting SeveralofSurge’sfeaturesrequireconnectivitytesting.Itmightbetestedinthreeways: ICMPPingTest:Asimplepingtestthatreflectsthecurrentstateofthephysicalnetwork. TherouterdelayintheMacversionofthehomecardandnetworkdiagnosticsaretheresultsofthistest. DNSLookupTest:ParallellookupoftheArecordforthebing.comdomaintoallDNSservers,resultingintheshortesttimearesponsewasreceived,reflectingcurrentphysicalnetworkconditions,withasimpleconfirmationofInternetaccess. DNSlatencyinthehomecardandnetworkdiagnostics,connectivitytestlatencyintheMainMenuforSurgeMacandthewidgetforSurgeiOSaretheresultsofthistest. HTTPtest:SendaHEADrequesttothetargetHTTPserver,calculatethetimetoreceivetheresponseheader,anyresponsepacketisconsideredtobevalid.ThetestURLcanbecustomized,itisrecommendedtochooseaURLwithnodesintheworld. Internetandproxylatencyinthehomecard,thebaselineforthepolicygroup,andtheproxytestfornetworkdiagnosticsaretheresultsofthistest. ThepolicygroupusesMethod3asthebaselineofjudgmentratherthanMethod1because: Theremaybearelayserverbetweentheproxyserver,andthepingtestcanonlyindicatethedelayinreachingtherelayserver. Inadditiontotestingtheconnectivitywiththeproxyserver,theDNSandegressnetworkconditionsoftheproxyservershouldalsobeconsidered. Someproxyprotocols,suchasSOCKS5,introduceadditionallatencyoverheadduetopoordesign,andshouldalsobeconsidered. 4.2.3.2url-testpolicygroup Testallsub-policiesconcurrentlyandselecttheonewiththelowestlatency.Thereareseveralparameters: url:TheURLfortesting. timeout:Themaximumtimetowaitforthetest,policiesexceedingthistimewillbemarkedasfailedandwillnotwaitanymore. interval:Thetimebetweentests.Thetesttimingforallurl-testclassgroupsis: Testonfirstuse. Triggersthetestagainiftheintervalbetweenthelasttestandsubsequentuseofthepolicygrouphasbeengreaterthantheintervalset.(Notethatthisdoesnotmeanthatthenexttestmustbeperformedeverynumberofminutes.Thepolicyneedstobeusedtotrigger) Thetestistriggereddirectlywhenthecurrentlyselectedpolicyproducesanunrecoverableerror. Afterthenetworkswitchover,theresultsoftheprevioustestwillbecleanedupandthefirsttestwillbetriggeredwhenthepolicygroupisused. tolerance:Ifthetestresultsofseveralpoliciesarenotverydifferent,itwillleadtofrequentswitchingamongthesepolicies,iftheexitIPsoftheproxyserversaredifferent,itmaytriggertheriskcontrolofthetargetwebsite.Therefore,atolerancedesignisaddedsothatswitchingtoanewpolicyisdoneonlywhenthedifferenceinlatencybetweenthebestpolicyandtheselectedpolicyisgreaterthanthetoleranceofthenewtestresult. evaluate-before-use:Bydefault,thefirstofthesub-policieswillbeuseddirectlywhenthepolicygroupisusedforthefirsttime,whiletriggeringthelatencytest.Ifevaluate-before-use=trueisconfigured,thenthefirstusewillwaituntilthetestiscompletebeforeselectingthebestpolicy. 4.2.3.3fallbackpolicygroup Thisisbasicallythesameastheurl-testgroup,withthedifferencethatyouonlycareaboutwhetherasub-policyisavailableandnotaboutthespecificlatency,andthenchoosethetoppolicyfromtheavailableones.Youcanmarkslowlinesasunavailableaswellbyturningdownthetimeoutparameter.Thereisnotoleranceparameterforthistype. 4.2.3.4load-balancepolicygroup Aload-balancinggroupisrandomlyselectedfromthesub-policiestouse. Whentheurlparameterisconfigured,availabilityischeckedagainstthebehaviorofthefallbackgroupandthenonlyarandomselectionismadefromtheavailablesub-policies. Inadditiontotheurl,timeout,andinterval,thereisoneotherparameter. persistent:whenpersistent=true,thesamepolicywillbeusedasmuchaspossibleforthesametargethostname.AvoidtriggeringriskcontrolsonthetargetsiteduetodifferentegressIPs.However,apolicychangemayoccurwhenavailabilitychanges. 4.2.3.5ssidPolicyGroup AlthoughstillcalledtheSSIDPolicyGroup,ithasbeenexpandedtoincludetheabilitytoselectsub-policiesbasedonthecurrentnetwork’sSSID,BSSID,routedIPaddress,etc.TheiOSversioncanalsospecifypoliciesfordatanetworks. 4.3Rules Surgeusesasystemofrulestomatchtheegresspolicyselectedforeachconnection.Therulesarematchedinatop-down,test-by-testprinciple.ThelastrulemustbeaFINALrule,usedwhenallrulesdonotmatch. 4.3.1DomainNameRules Matchesthisrulewhenthetargethostnamematches. DOMAIN:Matchesstrictlyadomainname. DOMAIN-SUFFIX:Matchesadomainnameanditssub-domains,e.g. DOMAIN-SUFFIX,apple.comwillmatchapple.comandwww.apple.com,butwillnotmatchanapple.com. DOMAIN-KEYWORD:Simplestringsearch,willmatchadomainnameifitcontainsasubstring. DOMAIN-SET:Designedforalargenumberofdomainnamelistfile,supportsfastsearchforthousandsofrecords.Eachlineinthefileisadomainname,ifalinebeginswith.matchesallsub-domainsandthedomainnameitself.Thiscanbeusedforadfiltering. 4.3.1.1Domainsandhostnames Thedomainnameisactuallyaformofthehostname,thereisnodistinctionbetweendomainandhostnamewithinSurge,allthedocumentationmentionsusethesamelogicforhandlingdomainnamesandhostnames. Forexample,theDOMAIN,1.2.3.4rulecanactuallybeusedtomatchconnectionswherethetargethostisanIPaddress1.2.3.4.DOMAIN,MacBook.localcanalsobeusedtomatchBonjourhostnames. 4.3.2IPaddressrule MatchestherulewhentheIPaddressofthetargethostmatches.Containsthreetypes:IP-CIDR,IP-CIDR6,andGEOIP. TheIPtyperuletriggerslocalDNSresolutionwhenthetargethostnameisadomainnameorhostname.JudgmentismadebasedontheIPaddressobtainedfromtheresolution.Whentheresolutionfails: *IfthefinalFINALrulehasadns-failedflag,thentheFINALrulewillmatchdirectly. *IftheFINALruledoesnothaveadns-failedtag,therequestwillsimplyfail. IPtyperuleshaveaproprietaryparameterno-resolve,ifanIPrulehasthisparameter,thenthe 1.Ifthetargethostnameisadomainname,theruleisskippedandDNSresolutionisnottriggered. 2.IfthetargethostnameisanIPaddress,followtherule. 3.IfthetargethostnameisadomainnameandanIPrulethatappearedearlierhasalreadytriggeredDNSresolutiontoobtaintheIPaddress,thenusethatIPaddresstomakeadecision. BecauseofthetimeoverheadofDNSlookups,whenconfiguringrules,itisoptimaltotrynottotriggerDNSresolutionfirstandtoplaceallrulesthatdotriggerDNSresolutionatthebottom.Thisway,requeststhatshoulduseproxypoliciesavoidlocalDNSresolution. However,thereisnoneedtointentionallyavoidresolutionaltogether,becauseonceyoudecidetousetheDIRECTpolicy,itwilleventuallyberesolved. Note,however,thatifatargethostnameisnotresolvableinthelocalDNS,youmustaddarulethatdeterminesthepolicytoterminatethematchbeforetriggeringtheDNS.Alternatively,addadns-failedflagtotheFINALruleanduseaproxypolicy. 4.3.3HTTPRelatedRules RulesthatarevalidonlyforHTTPrequests,includingURL-REGEXandUSER-AGENT. Specifically,URL-REGEXisnotvalidforunencryptedHTTPSconnections,onlybecausetheURLisavailableonlyafterMITMdecryption.However,theUSER-AGENTruleisvalidforunencryptedHTTPSconnectionsaswell,becauseprogramsusingtheHTTPproxywillsendaCONNECTrequestwiththeplaintextoftheirownUserAgent. 4.3.4OtherRules PROCESS-NAME:AvailableonlyforMacversion,canmatchprogramname. SRC-IP:MatchestheIPaddressofthesourceoftheconnection,canbeusedwhentakingoverconnectionsfromotherdevices. IN-PORT:Macversionsupportsmulti-portlistening,canconfigurespecificrulesfordifferentlisteningports. DEST-PORT:Canmatchtheportnumberofthetargethost. PROTOCOL:Canmatchaccordingtotheprotocoloftheconnection,therangeofvaluesisHTTP,HTTPS,TCP,UDP.(AlthoughlogicallyHTTPandHTTPSarebothspecialformsofTCP,therulewillbetreateddifferentlyaccordingtotheclassificationinthepreviouschapter.) SCRIPT:ApolicythatcanbechosencompletelyfreelybasedonvariousparametersusingJavaScript. 4.3.5Ruleset RULE-SETrulesetscanhavemultiplesub-rulesinasinglefileforeasysharingandreuse.However,therulesinarulesetcannotbepolicyspecific;theentirerulesetpointstoasinglepolicy. Inaddition,Surgecomeswithtworulesets,SYSTEMandLAN,andthespecificsub-rulescontainedintherulesetwillbeadjustedasSurgeisupdated.NotethattheLANrulesettriggersDNSresolution. 4.3.5.1DifferencebetweenRULE-SETandDOMAIN-SET RULE-SETcancontainalltypesofsub-rules,withnodifferenceinexecutionefficiencyfromtherulesinthemainconfiguration,whileDOMAIN-SETcanonlyusebothDOMAINandDOMAIN-SUFFIXformsofcontent,usingspeciallogicoptimizedtoprovideahugeperformanceboostwhenthereisverymuchcontent.(overathousanditems,otherwisethereisnotmuchdifferencebetweenthetwo) 4.3.6Logicrules AllruletypescanbecombinedbyusingAND,OR,NOToperations.Forexample: AND,((PROCESS-NAME,GoogleChrome),(PROTOCOL,UDP)),REJECT UDPpacketssentbyChromecanbedropped. Chapter5TLS,HTTPSandMITM ThischapterexplainsindetailtherelationshipbetweenTLSandHTTPS,theroleofTLS,andhowtoperformMITM. 5.1TheRelationshipbetweenTLSandHTTPS Thedesignofmoderncomputernetworkarchitectureadoptstheideaoflayering,andHTTPisanapplicationlayerprotocolbasedontheTCPprotocollayer. ThecorefunctionprovidedbyTCPlayerisreliabletransmission,andupperlayerapplicationsdonothavetoworryaboutpacketconstruction,splitting,disorder,packetlossandotherimplementationproblems. BasedonTCP,HTTPdefinesmoredetaileddatatransmissionstandardsandabstractsthedataflowintoasessionsystem,wheretheclientsendsarequestandtheserverrespondswitharesponse. HTTPSisaTLSlayerinsertedbetweentheTCPlayerandtheHTTPlayer,theTLSlayercanmakethedataflowoftheTCPlayerencryptedandsecurity.AllTCP-basedupperlayerprotocolscanbeprotectedinthiswaywithouthavingtoadjusttheprotocolitself.(e.g. SMTP) 5.2TheroleofTLS ItisgenerallythoughtthattheroleoftheTLSlayeristoencryptdata,butinfacttheTLSlayerhostsafullrangeofsecurityfeatures,includingatleast. Confidentiality:Akadataencryption,eveniftheattackergetsthecompletedatastreamfromthebeginningtotheendofthelink,theoriginaldatacannotbedecrypted. Authenticity:Itcanconfirmtheidentityofthetargethost.Forexample,whenaccessingexample.com,evenifthewholephysicalnetworkishijacked,itcanbeguaranteedthatitisthehostofexample.comandnototherimpostorhosts. Dataintegrity:Ensuresthatdatacannotbemodified,andthatanattackeronthelinkwhohasmodifiedthedatastreamwillcauseanerror. 5.2.1DataEncryptionandIntegrityProtection Insimpleterms,theTLShandshakephasegeneratesasymmetricencryptionkey(thesessionkey)throughasymmetricencryptionorkeynegotiationandexchangealgorithm,andthesubsequenttransmissionreliesonthesessionkeyforencryptionandintegrityprotection. 5.2.2Confirmingtheidentityofthetargethost BeforeweexplainhowTLSidentifiesthetargethost,weneedtoaddalittlebitofsimplecryptography:asymmetriccryptography.Insteadofdescribingthemathematicsofasymmetriccryptography,wewillsimplydescribehowitisused. 5.2.2.1AsymmetricEncryption Symmetricencryption,asthenamesuggests,referstothefactthatthesamekeyisusedforencryptionanddecryption.Asymmetricencryption,asthenamesuggests,referstothefactthattheencryptionanddecryptionusedifferentkeys,wecallthekeyusedforencryptionPublicKey,andthekeyusedfordecryptionPrivateKey.Ofcourse,onlytheprivatekeythatgoeswiththepublickeycandecryptthecontentencryptedbythepublickey,thispairofpublickeyandprivatekeyiscalledKeyPair.Thekeypairsatisfies: Itisnotpossibletocomputetheprivatekeydirectlyfromthepublickey. Ifthekeylengthmeetstherequirements,thearithmeticrequirementsforbrute-forcecrackingareunrealistic. Inadditiontobeingusedforencryptionanddecryption,thekeypaircanalsobeusedforsigning.Forapieceofcontent,theprivatekeycanbeusedtogenerateasignature(typicallycalledadigitalsignature),andthepublickeycanverifythatthesignaturewasgeneratedbytheprivatekeyasawaytomakesurethatthecontentisrecognizedbytheprivatekeyholder.Italsoensuresthatthecontentcannotbetamperedwith,andthatdigitalsignatureverificationwillfailifthecontentchanges. Themostcommonandwell-knownasymmetriccryptographicalgorithmistheRSAalgorithm,butthereareotherasymmetriccryptographicalgorithmsaswell. 5.2.2.2X.509CertificateChain Thenextstepistobrieflyunderstandwhatacertificateis,acertificateisactuallyadatastructureofkey-valuepairs(Key-Value),differentpurposeswillhavedifferentfieldsofcontent. Acertificatecorrespondstoakeypair,thepublickeyispartofthecertificate,whiletheprivatekeyiskeptprivatebytheownerofthecertificate. Acertificatecanbeissuedbyanothercertificate.Thecertificatecontainsadigitalsignaturefromtheparentissuingcertificateandinformationabouttheparentissuer. Thecertificateofthesuperiorissuercanbeissuedbyanothersuperiorissuertoformacertificatechain.Generally,TLSusesathree-levelcertificatechain,inwhichthehighestlevelissueriscalledtherootcertificate(orCAcertificate,CertificateAuthority)andtheintermediatecertificateiscalledIntermediateCertificate.Asfortheleafcertificate,wecallittheServerCertificatehere. TheoperatingsystemhaspresetmanyCAcertificates,whichmeansthattheoperatingsystemtruststheseCAcertificates,theintermediatecertificateistrustedbytheCAcertificatebypresentingthedigitalsignatureoftheCAcertificate,andtheservercertificateistrustedbytheintermediatecertificatebypresentingthedigitalsignatureoftheintermediatecertificate,whichconstitutesachainoftrust. Thenhowdoesthiscertificatetrustrelationshipcorrespondtotheidentityverificationintherealworld?Generallyspeaking,theCAcertificatemaintenanceauthorityneedstocomplywithspecificsecurityauditingandlaw-abidingoperation,andtheywillselectsomeagentstograntintermediatecertificates.Whenadomainnameholderwantstoobtainacertificateindicatinghisidentity,theagentwillfirstverifytheidentityoftheapplicant(e.g. throughthecontactemailofthedomainname)andthenissueacertificatetotheapplicant. TheabovesystemisabriefdescriptionoftheX.509certificatechain. Intheprocessofcertificateissuance,theapplicantusuallygeneratesthekeypair,thenencapsulatesthepublickeyasCSR(CertificateSigningRequest)andsendsittotheagent,andtheagentreturnsthecertificatewithdigitalsignatureofitsintermediatecertificateandtheapplicant’sdomainnameinthecertificatefieldafterconfirmingitsidentity.Theagentdoesnotknowtheprivatekeyofthecertificate. Theoperatingsystemcontinuallyadjuststhebuilt-inrootcertificatelibraryasthesystemisupdatedandaccordingtobusinessrequirements. Somesoftwarechoosestomaintainitsownrootcertificatestoreandignoresthesystem’srootcertificatestore,suchasFirefox. 5.2.2.3TLSHandshake Withtheabovebasicsinplace,wecanbegintoexplainhowTLSconfirmstheidentityofthetargethost.TakealookattheTLShandshakeinaction. Theclientexplicitlyinformstheserver,viaSNI,thatitisaccessingexample.comandasksfortheappropriatecertificate. Theserverconfirmsthatithasthecertificateforexample.comandprovidestheclientwithitsownservercertificateandallintermediatecertificates. TheclientreceivesthecertificateandconfirmsthattheCommonName(orSANfield)fieldofthecertificatecontainsexample.com. Confirmthattherootcertificatestorecanvalidatethecertificatethroughthecertificatechain. Wecannowconfirmthatthecertificateprovidedbytheserverisindeedanauthenticcertificateofexample.com. Theclientgeneratesarandompieceofcontent,encryptsitwiththepublickeyoftheservercertificate,andsendsittotheserver.Theservercanusetheprivatekeytodecryptthecontenttocalculatethesymmetricencryptionkeyforthesubsequenttransmissionphase,calledthesessionkey. Iftheserverdoesnothavetheprivatekeyforthiscertificate,thenitisimpossibletocalculatethecorrectsessionkeyandcontinuecommunicationwiththeclient. (TheaboveprocessissomewhatsimplifiedandmaybeslightlyinconsistentdependingontheTLSversionandencryptionmethod,readmoreathttps://www.cloudflare.com/learning/ssl/what-happens-in-a-tls-handshake/) Aftertheabovesteps,theclienthasfinallyestablishedasecurechannelwithexample.com.ItisnowreadyforfurtherHTTPcontenttransfer. 5.3MITMattack HavingexplainedhowTLSsecuresconnections,itwouldbetooeasytoexplainhowtodecryptTLStrafficviaMan-In-The-Middleattacks(MITM). Asthenameimplies,aMan-in-the-MiddleattackinterceptsplaintextcontentbyinsertingaMan-in-the-MiddleCbetweenclientAandserverB.ThisisdonebyhijackingtheconnectionfromAtoB.Theattackercanusethisman-in-the-middletechniquetointerceptTLStraffic.ThisisdonebyhijackingtheconnectionfromAtoBtoC,sothatAthinksitiscommunicatingwithBwhenitisactuallycommunicatingwithC.CthenestablishesaconnectiontoBandactsasago-betweentoforwardcontentbetweenAandB. ToimplementMITMtherearetwoconditions. CapableofhijackingA’snetwork,whichcanbeeasilydonebyproxyservices,VPNs,ISPs,Wi-Fiproviders,andotherlinkowners.(Malwarecanalsobeachievedbyhookingtheoperatingsystem,butifthemalwarehasobtainedsuchahighlevelofprivileges,thereisnoneedtouseMITMtodecryptthetraffic,justreadthememoryofthecorrespondingsoftware). NeedtobreakthroughtheTLStargethostidentitymechanism. Accordingtothepreviousdescription,sincewewanttoperformMITMforourownresearchpurposes,thewaytobreakthroughissimple:insertacertificatethatyouowntheprivatekeyintherootcertificatestoreofthesystem. 5.3.1Surge’sMITMprocess Let’stakeacompletelookattheprocessofSurgeperformingMITM. TheuserconfigurestheMITMfunction,Surgegeneratesthekeypairlocally,generatesarootcertificateandinstallsitintothesystemcertificatestore.TheMITMfunctionofexample.comisenabled. ReceiveaCONNECTrequesttoexample.com:433,enterMITMmode,anddirectlyinformclientthattheTCPhandshaketotheserverhasbeencompleted. TheclientbeginstheTLShandshakewithaClientHellomessage. Surgeimmediatelygeneratesaservercertificateforexample.comandsignstheconfiguredrootcertificate,andcompletesthehandshakewiththeclient. TheclientcommunicatesattheHTTPlayer,sendingarealHTTPrequest. AfterSurgereceivestherequest,itmodifiestherequestasdescribedinChapter2anddeterminesanoutboundpolicy.Usethecorrespondingpolicytoforwardtherequestbyinitiatingaconnectiontotherealexample.comandcompletingaTLShandshake. OnedetailthatvariesfromsoftwaretosoftwarewhenperformingMITMisthatSurgegeneratesanewservercertificateoutofairwhileithandshakingwiththeclient.Thestrategyofsomeothersoftwareistosuspendtheclienthandshake,starttheserverhandshakeimmediately,andaftercompletingtheserverhandshaketogettheservercertificate,modifyitspublickeyandissuerinformationandre-signwithitsownrootcertificate,andthenusethiscertificatetocompletethehandshakewiththeclient. ThereasonwhySurgeadoptsthefirstmethodisthatSurge’srulesystemallowstheegresspolicy(e.g.,URL)tobechosenbasedonHTTPlayercharacteristics,sothehandshakemustbecompletedfirsttogettheHTTPlayerrequestbeforeitcanestablishaconnectionwiththerealtargetserver. 5.3.2Publicrootcertificate Inpractice,wehavefoundthatsomeMITMtoolsdonotprovidealocallygeneratedrootcertificate.Instead,arootcertificateandtheprivatekeyofthecertificatearepackageddirectly. Thispracticeisveryinsecure,iftheusersystemtruststhecertificate,oncethenetworkhasbeenhijacked,attackerscanusethepubliccertificateoftheprivatekeyforMITMattackstodecrypttraffic. Besuretogenerateyourownuniquerootcertificatelocallyandkeeptheprivatekeysecurely. 5.3.3MITMAttackCountermeasures Asasoftwaredeveloper,ifyoudon’twantyourtraffictobedecryptedbyMITMtools,youneedtoperformMITMdefense. ThepurposeofX.509istoservethebrowser,whenusersvisitawebsite,theyknownothingexceptthedomainnameofthewebsite,sotheyhavetorelyonthecertificatechaintoverifythelegitimacyofthecertificate.However,thereisnosuchrestrictionforanapp.IntheTLShandshakephase,itisenoughtodirectlydeterminewhetherthepublickeyoftheservercertificateisapre-definedvalue,sothattheMITMtoolcannotbypasstheauthenticationanddecryptthetraffic. Inpractice,therearemanywaystodothis,suchascomparingthepublickeyoftheservercertificate.Iwon’tgointothemhere.IftheappperformsMITMdefenseandwantstoproceedwithMITM,youmustuseajailbrokendevicetomodifytheprogrambinaryorinjectruntimecodetobreakthroughthecustomcertificateverificationprocess. 5.4AdditionaldetailsofTLS We’lladdafewmoreTLS-relateddetailsinpassing. 5.4.1CommonHTTPSErrors The“insecure”errorsthatweoftenseeinbrowsingareerrorsthatresultfromfailuresatvariouspointsinthevalidationprocessdescribedabove,typically: Namedoesnotmatch:thepermitteddomainnameoftheservercertificateandtheURLbeingaccesseddonotmatch. Certificateexpires:Thecertificatehasanexpiredate,generallyforoneyear,theerrorgenerallyindicatesthatthewebsitemaintainerforgottoupdatethecertificate,ortheusersetthewrongsystemtime. Therootcertificatecannotbetrusted:therootcertificateofthecertificatechainprovidedbytheserverdidnotexistinthesystem’scertificatestore. Iftheaboveerrorisencounteredunderatrustednetwork(e.g. homebroadband),itisusuallycausedbyaconfigurationerrorbythewebmaster.Butiftheaboveerrorisencounteredonapublicnetwork,thenyouneedtobeextracareful,youmayhaveencounteredahijack. 5.4.2SNI WepreviouslymentionedSNIintheTLShandshake,sohere’sadetailedexplanationofwhatSNIdoes. Firstofall,weneedtoknowthatiftheserverhasanIPaddress11.22.33.44andtherearemultipledomainnamesexampleA.comandexmapleB.compointingtothisIP,whentheclientinitiatesaTCPconnectiontotheserver,theserverdoesnotknowwhethertheclientisaccessingthroughtheIPorexampleA.comorexmapleB.com,becausetheTCPmetadatacontainsonlytheIPaddress,andthedomainnameisnotintheTCP.ThedomainnameisusedbytheclienttoqueryfortheIPaddress,butdoesnotparticipateinthesubsequentTCPsessionsatall. ThisisaprobleminpracticewiththeHTTPprotocol,becauseofthescarcityofIPaddresses,wesometimeswantthesameIPaddress(orthesameserver)tobeabletoprovidedifferentcontent(i.e.,virtualhosts)dependingonthedomainnamethatthevisitorisaccessing.Tosolvethisproblem,thebrowseraddsaHostfieldintheHTTPrequestheader,whichisthehostnameportionoftheURL.TheservercanthenusetheHostfieldtodifferentiatebetweenthevisitedsitesandreturndifferentcontent. HTTPSalsoencountersthesameproblem,wheretheTLShandshakerequiresdifferentservercertificatesdependingonthedomainnamebeingaccessed.SincethecontentoftheHostexistsintheencryptedtransmissionafterthehandshake,ifthehandshakecan’tbecompleted,theproblemcannotbesolvedbythisfield.SoduringtheTLShandshake,theclient(browser)sendstheSNIofthedomainnametheclientisaccessinginplaintextfortheservertoselectthecertificate. Butthiscanleadtoprivacyleaks,allowingthelinkownertoknowthedomainnameofthesiteusersisvisiting.However,sincetheIPaddressaccessedbytheuserisalwaysknowable,itisnotreallyeasytosayhowserioustheimpactofthecompromiseofthevisiteddomainnameis. Inaddition,iftheTLSclientisabrowser,sincetheSNIofTLSandtheHostofHTTPheaderareboththehostnameoftheURL,theymustbeidentical.ButtheyarenotnecessarilyidenticalforotherTLSclients.Forexample,SurgesupportscustomizingtheSNIcontentoftheTLShandshake. 5.4.3ForwardSecrecy AsmentionedearlierwhenexplainingtheTLShandshake,thesessionkeyusedforTLSsubsequentsymmetricencryptioniscomputedfromrandomdataencryptedbytheprivatekeyoftheservercertificate. So,ifanattackersavesthesecretmessageofacommunication,andifsomedayinthefuturetheserver’sprivatekeyleaks,orifadvancesincomputersciencemakeitpossibletobrute-forcetheprivatekey,thesessionkeycanbeusedtodecrypttheprivatekey,thuscompletelydecryptingthesavedsecretmessage. Toaddressthisshortcoming,theTLSprotocolinusetodayisalittlemoresophisticatedinitshandshake,nolongersimplyusingstaticasymmetrickeypairstopasssessionkeys,butinsteadusingkeynegotiationalgorithmstogeneratetemporarysessionkeys.TheDHEalgorithm,forexample,isnowcommon. HereisabriefdescriptionoftheuseofDHE. 1.Theserverrandomlygeneratesakeypairforeachnewconnection:theserverprivatekeyandtheserverpublickey. 2.Theclientrandomlygeneratesakeypairforeachnewconnection:theclientprivatekeyandtheclientpublickey. 3.Theclientandserverexchangetheirpublickeys. 4.Theclientcalculatestheresult1byanalgorithmbasedontheserverpublickey,theclientprivatekey,andtheclientpublickey. 5.Theservercomputesresult2byanalgorithmbasedontheclientpublickey,theserverprivatekey,andtheserverpublickey. 6.Thealgorithmensuresthatresult1andresult2mustbeequal,andtheresultisusedtogeneratethesessionkey. 7.Theserverprivatekey,theclientprivatekey,result1,result2,andsessionkeyallexistonlyinthememoryofbothends,andarecompletelydiscardedandirretrievablewhentheconnectionisterminated. Insuchakeyexchangemechanism,noonecanrecalculatethesessionkeysincetheattackercanonlysavetheexchangedclientpublickeyandserverpublickey. ThisfeatureiscalledForwardSecrecy,orPerfectForwardSecrecy.TLSautomaticallychooseswhetherornottouseaforward-securedkeyexchangealgorithmduringthehandshake,dependingontheclientandserver. 5.4.4TLSCipherSuite Insummary,duringthehandshakephaseoftheTLSprotocol,theserverandclientneedtonegotiateseveraloutcomes TLSprotocolversion Keyexchangealgorithms Signaturealgorithms Symmetricencryptionalgorithms Hashalgorithm Themethodofnegotiationissimple:theclienttellstheserverwhatcombinationoftheabovefiveitemsitsupports,andtheserverchoosesthemostsecureresultitsupportsandtelltheclient.ThiscombinationiscalledTLSCipherSuite. TakeafewofthemostcommonTLSCipherSuitestodayasexamples. TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:ThisisacombinationofTLS1.2,thekeyexchangealgorithmisECDHE,thesignaturealgorithmisRSA,thesymmetriccryptographicalgorithmisAES-128-GCMandthehashalgorithmisSHA256. TLS_AES_256_GCM_SHA384:ThisisacombinationunderTLS1.3version.TheTLS1.3standardonlysupportskeyexchangeusingECDHEalgorithm,sononegotiationisneeded,andsincethecryptographicalgorithmsusedareallAEADalgorithmswiththeirownintegrityprotection,noseparateintegrityprotectionisneeded.Thesignaturealgorithmisalsonotsubjecttonegotiation.ThecombinationcontainsonlythesymmetricencryptionalgorithmasAES-256-GCMandthehashalgorithmasSHA384. IfyouareusingaTLS-basedproxyprotocol,youcanseetheCipherSuitenegotiatedbytheproxyconnectioninthenotestoSurge’sDashboard/RecentRequests. Chapter6DNS Surgedoesn’tusethesystemDNSresolutionentirelyandimplementsitallbyitself. 6.1QueryInParallel SurgeperformsDNSlookupstoallconfiguredDNSupstreamserverssimultaneouslyandselectsthefastestreturningresulttoimproveperformance.Thisfeatureisconsistentwiththednsmasqimplementation. 6.2OptimisticDNS Duetothecomplexityofmodernnetworks,mostsitesconfiguretheirDNSrecordlifetime(TTL)foraveryshortperiodoftime,suchas30seconds.ThisallowsnetworkadministratorstomakechangestoDNSrecordsthattakeeffectquickly,withouthavingtowaitforallnodestotimeoutofTTLfortroubleshootingandmaintenance. Understandably,websiteandAPIavailabilityisatoppriorityformanycompanies.IfanIPisunreachable,itcantakeupto24hoursforaDNSrecordtotakefulleffectaftermodifyingit,causingincalculabledamage.SooperationsandmaintenancearelikelytochooseaveryshortTTL. Butthisbringsaproblem,theclientwillstrictlyfollowtheTTLtoperformaquery,soeveryshorttimewillbealookupagain,aDNSquerytimecostisasshortasafewmilliseconds,butthelongestcantakeseveralseconds.Frequentrepetitionoflookupscausesunnecessarydelays. That’swhyAppleintroducedOptimisticDNSatWWDC2018,whichallowstheprogramtocontinuetouseoldresultswhileperformingthenewDNSlookupswhenestablishingaconnectionifthelocalDNScachehasexpired,andthentryagainwithnewresultsiftheconnectionfails. Inmostcases,theDNSrecordsareunchanged,sothissolutionwillnotaffectnormaluseatall.WhenencounteringaDNSrecordupdate(theprobabilityisusuallysmall),itwillonlydelayoneortworequests.Thiscanbesaidaperfectoptimization. However,duetolimitationssuchasPOSIXstandard,Appledoesnotapplythisoptimizationeverywhere,butSurgedoes,soallrequestscanenjoyOptimisticDNSoptimization. 6.3LocalMapping SurgesupportstheconfigurationoflocalDNSmappings,whichfunctionbasicallythesameasthe/etc/hostsfile.InadditiontodirectlyspecifyingtheIPaddressofahostname,SurgealsosupportscustomizingtheDNSserverforaspecificdomain.Oryoucancompletelycustomizetheresolutionlogicviascript. 6.4Usingsystem’sresolution SurgesupportsconfiguringsomedomainnamestofallbacktosystemDNSresolution(example.com=server:syslib)toresolvecompatibilityissues,suchassomeVPNsusetheSplitDNSmechanismtoaddaDNSserveronthesystemtohandlespecificdomainnames.ThiscomplexlogiccanbesolvedbyconfiguringfallbackonVPN-relateddomainnames.