<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Santanu Sinha]]></title><description><![CDATA[Data Science & AI/ML — fundamentals, enterprise best practices, and the latest innovations.]]></description><link>https://www.datascienzz.com</link><image><url>https://substackcdn.com/image/fetch/$s_!fqGm!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5eacf7a2-82fa-4e20-982a-36b07bbbf81b_500x500.png</url><title>Santanu Sinha</title><link>https://www.datascienzz.com</link></image><generator>Substack</generator><lastBuildDate>Sun, 12 Apr 2026 06:14:02 GMT</lastBuildDate><atom:link href="https://www.datascienzz.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Santanu Sinha]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[datascienzz@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[datascienzz@substack.com]]></itunes:email><itunes:name><![CDATA[Santanu Sinha]]></itunes:name></itunes:owner><itunes:author><![CDATA[Santanu Sinha]]></itunes:author><googleplay:owner><![CDATA[datascienzz@substack.com]]></googleplay:owner><googleplay:email><![CDATA[datascienzz@substack.com]]></googleplay:email><googleplay:author><![CDATA[Santanu Sinha]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Let's Build an OpenAI-style Text-to-Token Visualizer]]></title><description><![CDATA[How GPTs turn any text into universal tokens with UTF-8 & Byte Pair Encoding (BPE)]]></description><link>https://www.datascienzz.com/p/text-to-token-visualizer</link><guid isPermaLink="false">https://www.datascienzz.com/p/text-to-token-visualizer</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Fri, 22 Aug 2025 12:53:34 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!_a5b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_a5b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_a5b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!_a5b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!_a5b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!_a5b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!_a5b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png" width="1024" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:608,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_a5b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!_a5b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!_a5b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!_a5b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62440231-cca0-48b5-8fbc-afd6cfe0685e_1024x608.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p>In my previous <a href="https://www.datascienzz.com/p/how-gpt-reads-your-message-byte-by">article</a>, I talked about how GPTs process text into tokens.</p><p>Here, I have created a simple <a href="https://github.com/sinha505/tokenizer-mvp">implementation</a> of the logic - in <a href="https://platform.openai.com/tokenizer">OpenAI style</a>. </p><p>Specifically, a lightweight Streamlit app that shows how text is broken down into tokens - the basic building blocks that GPT models use to understand language. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XGxP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XGxP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XGxP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XGxP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XGxP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!XGxP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg" width="638" height="371.8863503222027" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:995,&quot;width&quot;:1707,&quot;resizeWidth&quot;:638,&quot;bytes&quot;:109231,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171651671?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e15b358-b960-43ba-85af-ff85e2cc6578_1707x995.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XGxP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XGxP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XGxP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XGxP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f4bc2be-82b3-46ba-8e40-0ed074e4e590_1707x995.jpeg 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The Streamlit-based UI</figcaption></figure></div><p>Simply paste any input, pick a tokenizer (cl100k_base, gpt2, p50k_base, r50k_base), and generate tokens, token IDs, and UTF-8 bytes. </p><p>The results can vary depending on the tokenizer that is used. I found that cl100k_base is similar to GPT&#8209;4/3.5. </p><p>The Python implementation is available on <a href="https://github.com/sinha505/tokenizer-mvp">GitHub</a>.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Subscribe for more.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/text-to-token-visualizer/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/text-to-token-visualizer/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[How GPT Seamlessly Handles Multiple Languages, Math Notations, and Emojis]]></title><description><![CDATA[How GPTs Convert Text to Universal Tokens with UTF-8 & Byte Pair Encoding (BPE)]]></description><link>https://www.datascienzz.com/p/how-gpt-reads-your-message-byte-by</link><guid isPermaLink="false">https://www.datascienzz.com/p/how-gpt-reads-your-message-byte-by</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Thu, 21 Aug 2025 23:35:08 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Gdzl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Gdzl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Gdzl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!Gdzl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!Gdzl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!Gdzl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Gdzl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png" width="1024" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:608,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Gdzl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!Gdzl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!Gdzl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!Gdzl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3392915b-916d-4bfe-9a4c-7d6692e465df_1024x608.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p>Ever wondered how models like <strong>ChatGPT</strong> read your message and respond so well?</p><p>It all starts with how text is converted into something called a <strong>token</strong>.</p><p>This post will walk you through exactly how GPT breaks down your sentence - from raw characters to 8-bit bytes to token IDs - with <strong>real code examples</strong>.</p><div class="pullquote"><p>Let&#8217;s say you type: "hello <em>&#128522;</em>"</p><p>What happens inside GPT?</p></div><h3>Step-1: Text to UTF-8 Encoding to 8-bit Bytes</h3><p>Every character of your text is first turned into <strong>UTF-8 bytes,</strong> i.e., 8-bit numbers in the range 0&#8211;255. So, when you type "hello &#128522;", the following is generated:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hC2e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hC2e!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hC2e!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hC2e!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hC2e!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!hC2e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg" width="592" height="158.26481715006304" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:212,&quot;width&quot;:793,&quot;resizeWidth&quot;:592,&quot;bytes&quot;:52215,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171573379?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hC2e!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hC2e!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hC2e!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hC2e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90f8b686-dbe6-41e8-87de-a11e72870be7_793x212.jpeg 1456w" sizes="100vw"></picture><div></div></div></a></figure></div><p>Try the snippet below in Python:</p><pre><code>text = "hello &#128522;"
utf8_bytes = text.encode("utf-8")
print(list(utf8_bytes))

Output: [104, 101, 108, 108, 111, 32, 240, 159, 152, 138]</code></pre><p>Your text is converted to a byte stream as shown above, where each number is further converted to an 8-bit chunk. Emojis and non-English characters can take multiple bytes.</p><h3>Step-2: Byte Pair Encoding (BPE)</h3><p>Once we have byte sequences, GPT applies a clever trick called <strong>Byte Pair Encoding (BPE)</strong>. It is a smart compression trick to merge frequent byte patterns into homogeneous groups. </p><p>How?</p><blockquote><ul><li><p>It finds frequent adjacent byte pairs and merges them into reusable chunks called tokens.</p></li><li><p>It continues merging to create a vocabulary of common chunks (like "the", "##ing", ".com", "&#128514;", etc.)</p></li></ul></blockquote><p><strong>Example:</strong></p><ul><li><p>('l', 'l') &#8594; 'll'  </p></li><li><p>('e', 'll') &#8594; 'ell'  </p></li><li><p>('h', 'ell') &#8594; 'hell'  </p></li><li><p>('hell', 'o') &#8594; 'hello' and so on.</p></li></ul><p>You can use OpenAI&#8217;s tokenizer (<em>tiktoken</em>) to see this in action:</p><pre><code>import tiktoken
enc = tiktoken.get_encoding("gpt2")       # other tokenizer: "cl100k_base"
tokens = enc.encode("hello &#128522;")
print(tokens)                             # Token IDs
print([enc.decode([t]) for t in tokens])  # Token strings

Output:
[15496, 220, 96050]
['hello', ' ', '&#128522;']</code></pre><p>So, here GPT replaces:</p><blockquote><ul><li><p>"hello" with token 15496</p></li><li><p>" "  with  token 220</p></li><li><p>"&#128522;" with token 96050</p></li></ul></blockquote><p>These are tokens, not characters or bytes anymore.</p><h3>Step-3: Assign Each Token a Number (ID)</h3><p>GPT doesn't store text - it stores token IDs. </p><p>So ['hello', ' ', '&#128522;'] is represented as [15496, 220, 96050].</p><p>Token IDs depends on the specific tokenizer vocabulary. </p><p><strong>The above byte-to-token process is efficient as:</strong></p><ul><li><p>Handles all languages and emojis</p></li><li><p>Takes fewer tokens than characters or words</p></li><li><p>Compact and finite vocab </p></li></ul><h4>Final Output:</h4><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fmxr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fmxr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fmxr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fmxr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fmxr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!fmxr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg" width="692" height="66.77836691410393" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:91,&quot;width&quot;:943,&quot;resizeWidth&quot;:692,&quot;bytes&quot;:32469,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171573379?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fmxr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fmxr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fmxr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fmxr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe70f72f0-956b-4dde-ba2d-7639d75da2b5_943x91.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><h3>Step-4: Token IDs as Model Input</h3><p>These token IDs are now passed into the GPT model for further processing, learning, or generation. </p><div class="pullquote"><p>So, input to model would be: [15496, 220, 96050]</p></div><p>That&#8217;s all GPT needs to predict the next word, translate, summarize, or write code.</p><h3>Token Vocabulary: Pre-Built and Reused</h3><p>Before training, GPT builds a fixed vocabulary of frequent byte patterns using BPE on large text datasets. This vocabulary and merge rules are saved and not updated later.</p><p>During inference, GPT:</p><blockquote><ul><li><p>Converts text to UTF-8 bytes</p></li><li><p>Applies the same BPE merge rules</p></li><li><p>Uses the same token vocabulary</p></li><li><p>Merging stops when no further valid merge is found in the pre-built dictionary.</p></li></ul></blockquote><p>This ensures consistent tokenization for all inputs.</p><p>Different GPT models use different tokenizers with fixed vocabulary sizes. For example, GPT-2 and GPT-3 use BPE with ~50K tokens, GPT-4 uses a newer BPE with 100K+, and LLaMA uses SentencePiece with ~32K tokens.</p><h3>What Happens During Inferencing?</h3><blockquote><ul><li><p><strong>First, universal tokens are generated during training and stored in a fixed vocabulary. </strong></p></li><li><p><strong>Later, when you provide text, the tokenizer matches it against this vocabulary and represents it as a sequence of token IDs. </strong></p></li></ul></blockquote><p>Below is a Python implementation of the above logic during inference:</p><pre><code>import tiktoken

# Use a tokenizer  #gpt2, cl100k_base
enc = tiktoken.get_encoding("gpt2")  
 
# Input sentence
text = "machine learning is fun!"

# Tokenize using BPE
tokens = enc.encode(text)
decoded = [enc.decode([t]) for t in tokens]

# Display
print("Original text:", text)
print("Token IDs:", tokens)
print("Tokens:", decoded)</code></pre><p>You can try the above code to tokenize your own texts.</p><h3>Why Can&#8217;t ASCII Do This?</h3><p>ASCII is a 7-bit system from the 1960s. </p><p>It works for English A&#8211;Z, a&#8211;z, digits, and punctuation.</p><p>But it breaks when you use:</p><ul><li><p>Emojis like &#128526;</p></li><li><p>Accented letters like &#8220;&#233;&#8221;</p></li><li><p>Non-English scripts like Hindi, Chinese, Arabic</p></li></ul><p><strong>Example:</strong></p><pre><code>text = "&#2344;&#2350;&#2360;&#2381;&#2340;&#2375;"
print(text.encode("ascii"))  # &#128680;

Output:
UnicodeEncodeError: 'ascii' codec can't encode characters...</code></pre><p>But with UTF-8:</p><pre><code>print(list(text.encode("utf-8")))

Output:
[224, 164, 168, 224, 164, 174, 224, 164, 184, 224, 165, 141, 224, 164, 164, 224, 165, 135]</code></pre><h3>Summary</h3><p>So, your text is turned into <strong>UTF-8 bytes</strong>, merged into <strong>tokens with BPE</strong>, and mapped to <strong>token IDs</strong>. </p><p>That&#8217;s why it works across all languages, symbols, and emojis and is truly language-agnostic and internet-friendly.</p><p>ChatGPT doesn&#8217;t actually read words or emojis. It reads <strong>numbers </strong>using the available tokens. Because of the universal tokenizer, it&#8217;s possible to represent all possible language texts, emojis, and even math notations.  And for that rare or unusual text(s) it will take more tokens to represent.</p><blockquote><p><em>Finally:  Text --&gt; UTF-8 bytes --&gt; BPE merges --&gt; Tokens --&gt; Token IDs.</em></p></blockquote><p>A Python implementation is shown <a href="https://www.datascienzz.com/p/text-to-token-visualizer">here</a>. The code is also available in <a href="https://github.com/sinha505/tokenizer-mvp">GitHub</a>.</p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading! Subscribe for more.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/how-gpt-reads-your-message-byte-by/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/how-gpt-reads-your-message-byte-by/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[Cross-Validation: Techniques, Pitfalls, and Best Practices]]></title><description><![CDATA[Why Cross-Validation is the lie detector of your ML model...]]></description><link>https://www.datascienzz.com/p/cross-validation-techniques-pitfalls</link><guid isPermaLink="false">https://www.datascienzz.com/p/cross-validation-techniques-pitfalls</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Mon, 18 Aug 2025 05:52:44 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!YgKw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YgKw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YgKw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!YgKw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!YgKw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!YgKw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YgKw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png" width="1024" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/eba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:608,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!YgKw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!YgKw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!YgKw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!YgKw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feba1d6a9-fde0-4ba5-a261-23f4e2809d31_1024x608.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><h3>What is Cross-Validation and Why it Matters</h3><p><strong>Cross-Validation (CV) is a method to test your model&#8217;s honesty. </strong></p><p>A single train-test split can fool you. It might make a weak model look good by chance. CV fixes this by evaluating performance across multiple data splits, giving a more reliable estimate.</p><ul><li><p><strong>Overfitting?</strong> High train score and low CV score. Means high variance.</p></li><li><p><strong>Underfitting?</strong> Both scores low. Means high bias.</p></li></ul><p>Think of it as a stress test for your model - revealing how well it performs in different scenarios.  And evaluating performance across CV folds, you can pick the best model or hyperparameters during model training.</p><div class="pullquote"><p>After all, one split can lie. Cross-validation doesn&#8217;t.</p></div><h3><strong>How Cross-Validation Works</strong></h3><p>Below is a pictorial representation of how CV works (Source: <em>Max Kuhn, Applied Predictive Modeling</em>):</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3GPk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3GPk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 424w, https://substackcdn.com/image/fetch/$s_!3GPk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 848w, https://substackcdn.com/image/fetch/$s_!3GPk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 1272w, https://substackcdn.com/image/fetch/$s_!3GPk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!3GPk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf" width="523" height="165.7206132879046" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:186,&quot;width&quot;:587,&quot;resizeWidth&quot;:523,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3GPk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 424w, https://substackcdn.com/image/fetch/$s_!3GPk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 848w, https://substackcdn.com/image/fetch/$s_!3GPk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 1272w, https://substackcdn.com/image/fetch/$s_!3GPk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba37693b-9d7c-487d-92bf-8429bd1f9c6b_587x186.emf 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Average the performance metrics across prediction folds. The result is <strong>an unbiased estimate</strong> of generalized error assuming no leakage. </p><p>And how does it work along with parameter selection in a grid search? </p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!AeEZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!AeEZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 424w, https://substackcdn.com/image/fetch/$s_!AeEZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 848w, https://substackcdn.com/image/fetch/$s_!AeEZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!AeEZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!AeEZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg" width="524" height="231.8266818700114" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:388,&quot;width&quot;:877,&quot;resizeWidth&quot;:524,&quot;bytes&quot;:72652,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!AeEZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 424w, https://substackcdn.com/image/fetch/$s_!AeEZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 848w, https://substackcdn.com/image/fetch/$s_!AeEZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!AeEZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60407ccd-2891-4295-8d34-8b84c29eb12b_877x388.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><h4><strong>Illustration</strong></h4><p>Say you want to try three possible values of a hyper-parameter m = [20, 30, 40]. After a 3 fold CV, the below results are obtained:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!h8CT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!h8CT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 424w, https://substackcdn.com/image/fetch/$s_!h8CT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 848w, https://substackcdn.com/image/fetch/$s_!h8CT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!h8CT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!h8CT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg" width="484" height="93.3938706015891" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:170,&quot;width&quot;:881,&quot;resizeWidth&quot;:484,&quot;bytes&quot;:42018,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!h8CT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 424w, https://substackcdn.com/image/fetch/$s_!h8CT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 848w, https://substackcdn.com/image/fetch/$s_!h8CT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!h8CT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F360b31ea-fa53-4e86-acdc-737a69c61162_881x170.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Selected <strong>m = 30 </strong>with best cross-validated performance.</p><h4><strong>Where CV Helps and Where it Doesn&#8217;t</strong></h4><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!K-is!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!K-is!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 424w, https://substackcdn.com/image/fetch/$s_!K-is!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 848w, https://substackcdn.com/image/fetch/$s_!K-is!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!K-is!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!K-is!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg" width="598" height="117.89837398373983" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:194,&quot;width&quot;:984,&quot;resizeWidth&quot;:598,&quot;bytes&quot;:51397,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a11bbae-786a-4deb-baf6-548b41471558_985x202.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!K-is!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 424w, https://substackcdn.com/image/fetch/$s_!K-is!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 848w, https://substackcdn.com/image/fetch/$s_!K-is!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!K-is!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F641e8a7d-1f09-4900-9284-df737138ac89_984x194.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><h4>Why CV is Rare in Deep Learning</h4><ul><li><p>Training a deep neural network with millions of parameters <em>K</em> times is super expensive due to the high compute cost.</p></li><li><p>The common approach is to use a train/validation/test split with early stopping.</p></li><li><p>Also, it&#8217;s a good practice to keep a final unseen test set.</p></li></ul><h4><strong>Common CV Methods</strong></h4><p>There are several types of CV methods as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8GKK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8GKK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 424w, https://substackcdn.com/image/fetch/$s_!8GKK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 848w, https://substackcdn.com/image/fetch/$s_!8GKK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!8GKK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!8GKK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg" width="692" height="397.6523517382413" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:562,&quot;width&quot;:978,&quot;resizeWidth&quot;:692,&quot;bytes&quot;:142265,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae1cc7c4-040f-4f50-b53c-675863301aab_986x567.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8GKK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 424w, https://substackcdn.com/image/fetch/$s_!8GKK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 848w, https://substackcdn.com/image/fetch/$s_!8GKK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!8GKK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d82577-f61d-4528-9071-317f9f074358_978x562.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><h3><strong>A Few Important Things to Note</strong></h3><h4>1. CV and Bias/Variance</h4><p>Cross-validation (CV) doesn&#8217;t reduce bias or variance. It simply gives you a more <strong>unbiased estimate of total error (MSE).</strong></p><p>Let&#8217;s break it down:</p><blockquote><ul><li><p><strong>Bias:</strong> Error from overly simple assumptions (e.g., fitting a straight line to a curve).</p></li><li><p><strong>Variance:</strong> Error from model instability i.e. small data changes lead to large prediction shifts.</p></li></ul></blockquote><p>The total expected error is:</p><blockquote><p><strong>Total Error (MSE) = Bias&#178; + Variance + Irreducible Noise</strong></p></blockquote><p><strong>Let&#8217;s see an example:</strong></p><ul><li><p>True value = 50</p></li><li><p>Model predictions on 3 resampled datasets = 42, 56, 50</p></li><li><p>Average prediction = 49.33</p></li><li><p>Bias&#178; = (49.33 - 50)&#178; = 0.44</p></li><li><p>Variance = mean squared deviation from the average = (1/n) &#215; sum[(x&#7522; &#8722; x&#772;)&#178;]</p><p>Or, Variance = [(42 - 49.33)<sup>2 </sup>+ (56 - 49.33)<sup>2 </sup>+ (50 - 49.33)<sup>2</sup>] / 3 = 32.89</p></li><li><p>Total error = 33.33</p></li></ul><blockquote><p><em>CV with grid search evaluates several model parameters to identify the configuration with the lowest cross-validated error. So, CV is rather a model selection process and not an error minimization algorithm in itself.</em></p></blockquote><blockquote><p>Now, the standard k-fold CV won&#8217;t give us bias&#178; and variance separately, since each point is predicted once. To estimate these components, we need multiple predictions per point, like repeated CV or bootstrapping.</p></blockquote><h4>2. Example of 3-Fold Nested CV:</h4><p>Imagine the data is split into three parts: D1, D2, and D3.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XMBR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XMBR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XMBR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XMBR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XMBR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!XMBR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg" width="495" height="90.22998296422487" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:107,&quot;width&quot;:587,&quot;resizeWidth&quot;:495,&quot;bytes&quot;:22182,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XMBR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XMBR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XMBR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XMBR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F823a2993-fce2-43ae-a6e9-47c2c2663ecc_587x107.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Inner CV fold is used to tune while the outer CV folds give unbiased performance estimate, unlike the regular K-Fold CV. </p><h4>3. Where We Need Group-K-Fold</h4><p>Imagine you're building a model to predict customer churn, and each customer has multiple transactions:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-dYV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-dYV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-dYV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-dYV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-dYV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!-dYV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg" width="393" height="110.20206185567011" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:136,&quot;width&quot;:485,&quot;resizeWidth&quot;:393,&quot;bytes&quot;:19218,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-dYV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-dYV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-dYV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-dYV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa288061e-9ea9-40df-8414-d311b5978b58_485x136.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>If you use regular k-Fold, T1 might go to training and T2 to validation and leaking customer behaviour. <strong>Group-K-Fold</strong> keeps all of customer A's data together, preserving independence between folds and avoiding data leakage during cross-validation.</p><p>Other similar examples are shopping sessions, patient records, sensor batch, and user ID-specific activities. In such cases, <strong>Group-K-Fold </strong>is used to keep multiple data points belonging to the same entity together. </p><h4><strong>4. Picking the Right CV Strategy</strong></h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!OB-U!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OB-U!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 424w, https://substackcdn.com/image/fetch/$s_!OB-U!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 848w, https://substackcdn.com/image/fetch/$s_!OB-U!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!OB-U!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!OB-U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg" width="656" height="271.42433862433865" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:391,&quot;width&quot;:945,&quot;resizeWidth&quot;:656,&quot;bytes&quot;:106325,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!OB-U!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 424w, https://substackcdn.com/image/fetch/$s_!OB-U!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 848w, https://substackcdn.com/image/fetch/$s_!OB-U!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!OB-U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6671695f-6290-438e-88d9-89bd6a03bd61_945x391.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>5. Common Pitfalls and the Fix</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cYl0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cYl0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 424w, https://substackcdn.com/image/fetch/$s_!cYl0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 848w, https://substackcdn.com/image/fetch/$s_!cYl0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!cYl0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!cYl0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg" width="674" height="390.6624203821656" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:546,&quot;width&quot;:942,&quot;resizeWidth&quot;:674,&quot;bytes&quot;:97772,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.datascienzz.com/i/171204645?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!cYl0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 424w, https://substackcdn.com/image/fetch/$s_!cYl0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 848w, https://substackcdn.com/image/fetch/$s_!cYl0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!cYl0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99f79789-ce72-402a-8296-e03f0f8a35aa_942x546.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>6. Why Decision Trees Have High Variance </h4><p><strong>Decision trees use greedy, local splits that are highly sensitive to small data changes. </strong>A minor variation like a new row or an outlier - can change the first split and ripple through the entire tree, leading to drastically different models. </p><blockquote><p><strong>Since they don&#8217;t globally optimize or backtrack, this recursive structure makes them unstable and prone to overfitting, especially with deep trees.</strong></p></blockquote><blockquote><p><strong>Using Random Forests or other ensemble methods helps address the variance problem. </strong></p></blockquote><p>By aggregating predictions from multiple trees trained on different data subsets, these models stabilize performance and reduce sensitivity to small data changes.</p><h4>7. Some Best Practices &amp; Takeaways</h4><ul><li><p>Cross-validation improves the model evaluation process but cannot guarantee generalization.</p></li><li><p>Always do pre-processing inside the CV loop - otherwise there may be data leakage.</p></li><li><p>Be leakage-aware: no scaling, encoding, or feature selection before CV.</p></li><li><p>For tuning, nested CV works better.</p></li><li><p>Use Stratified-K-Fold, Time-Series-Split, and Group-K-Fold where appropriate as mentioned above.</p></li></ul><h3>Reference</h3><ol><li><p>Kuhn, M., &amp; Johnson, K. (2013). Applied Predictive Modeling. Springer.</p></li><li><p>Hastie, T., Tibshirani, R., &amp; Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer.</p></li><li><p>Chollet, F. (2021). Deep Learning with Python (2nd Edition). Manning.</p></li></ol><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading! Subscribe for more.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/cross-validation-techniques-pitfalls/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/cross-validation-techniques-pitfalls/comments"><span>Leave a comment</span></a></p><p></p><p></p>]]></content:encoded></item><item><title><![CDATA[Why We Need Matrices & GPUs for Deep Learning]]></title><description><![CDATA[A simple illustration of how matrices and GPUs work together...]]></description><link>https://www.datascienzz.com/p/why-we-need-matrices-and-gpus-for</link><guid isPermaLink="false">https://www.datascienzz.com/p/why-we-need-matrices-and-gpus-for</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Mon, 11 Aug 2025 04:13:26 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!6jBr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6jBr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6jBr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!6jBr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!6jBr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!6jBr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!6jBr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png" width="1024" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:608,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!6jBr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!6jBr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!6jBr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!6jBr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7df5e4b8-c891-43df-8223-ab1e5a3b34a4_1024x608.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><blockquote><p>Deep learning rs on massive neural networks with billions of parameters, especially in fields like computer vision, NLP, and LLMs. This raises a key question: how are such large computations handled efficiently? </p></blockquote><p>In this article, we&#8217;ll use simple examples to show how matrices and GPUs make it possible &#8212; leveraging matrix properties like additivity and associativity.</p><p>So, we already know that deep learning consists of massive neural networks built with varying architectures. With the rise of ground-breaking applications in computer vision, natural language processing (NLP), and large language models (LLMs), the scale of such neural networks has become unimaginable and often to the tune of billion parameters.</p><div class="pullquote"><p>This raises an important question: How exactly do we handle such massive computations efficiently?</p></div><p>In this article, we will take an insider look at how exactly matrices and GPUs (Graphics Processing Units) help handling such large-scale deep learning models.</p><p>Please note that, for illustration, a massive real-life neural network will be difficult to imagine. Hence, we will use only a very simple neural network just to drive the concept. So, let&#8217;s understand the mechanism.</p><h3><strong>ILLUSTRATION</strong></h3><p>Let us consider the data below that captures how Social-Media and Google ad-spend influence customer conversions. We want to train a neural network on it.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5_JY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5_JY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 424w, https://substackcdn.com/image/fetch/$s_!5_JY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 848w, https://substackcdn.com/image/fetch/$s_!5_JY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!5_JY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!5_JY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg" width="467" height="234.69335604770018" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:295,&quot;width&quot;:587,&quot;resizeWidth&quot;:467,&quot;bytes&quot;:23709,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5_JY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 424w, https://substackcdn.com/image/fetch/$s_!5_JY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 848w, https://substackcdn.com/image/fetch/$s_!5_JY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!5_JY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3330d5b6-d85b-4beb-a05d-d923cc7501a9_587x295.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Let us consider the world&#8217;s simplest Neural Network and see how it is trained on the above data. To keep it even simpler, we ignore any hidden layer, activation function, and bias. However, the concept works for all large scale complex neural models.</p><p>Since there are two inputs (ad-spends) and one output (conversions), we consider two input neurons.</p><p>Consider two weight factors as below - i.e. inputs with random initial values <em>w</em>1 = 0.5 and <em>w</em>2 = 0.7:</p><p>Assume learning rate i.e., a small step size for updates:</p><blockquote><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!p5sC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!p5sC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 424w, https://substackcdn.com/image/fetch/$s_!p5sC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 848w, https://substackcdn.com/image/fetch/$s_!p5sC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 1272w, https://substackcdn.com/image/fetch/$s_!p5sC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!p5sC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png" width="76" height="23" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:23,&quot;width&quot;:76,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!p5sC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 424w, https://substackcdn.com/image/fetch/$s_!p5sC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 848w, https://substackcdn.com/image/fetch/$s_!p5sC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 1272w, https://substackcdn.com/image/fetch/$s_!p5sC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F762b4418-f83a-4bfd-8f92-edab87d368a9_76x23.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div></blockquote><p>Now, let&#8217;s take the first row of the data as inputs <em>x</em>1 = 2 and <em>x</em>2 = 5 and the target output <em>y</em> = 10.</p><p>We will do the calculations for forward pass and backpropagation to adjust weights for the first row of the training data set.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!q6-4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!q6-4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 424w, https://substackcdn.com/image/fetch/$s_!q6-4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 848w, https://substackcdn.com/image/fetch/$s_!q6-4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!q6-4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!q6-4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg" width="350" height="196.94915254237287" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:332,&quot;width&quot;:590,&quot;resizeWidth&quot;:350,&quot;bytes&quot;:25750,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!q6-4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 424w, https://substackcdn.com/image/fetch/$s_!q6-4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 848w, https://substackcdn.com/image/fetch/$s_!q6-4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!q6-4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc44b68ed-1e70-4d17-96ca-0fe0031ab7d8_590x332.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>We will do the calculations first in simple manual steps:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dVcN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dVcN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 424w, https://substackcdn.com/image/fetch/$s_!dVcN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 848w, https://substackcdn.com/image/fetch/$s_!dVcN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!dVcN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!dVcN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg" width="560" height="649.3421052631579" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:705,&quot;width&quot;:608,&quot;resizeWidth&quot;:560,&quot;bytes&quot;:89360,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dVcN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 424w, https://substackcdn.com/image/fetch/$s_!dVcN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 848w, https://substackcdn.com/image/fetch/$s_!dVcN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!dVcN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe054a692-5cad-49a8-9f0c-3212576c39bc_608x705.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>I must say, this is a highly trivial process and does not scale up for more complex settings! Now, let us see how we can put a structure around it for scalability and easy handling. Ideally the matrices way!</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dH7P!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dH7P!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 424w, https://substackcdn.com/image/fetch/$s_!dH7P!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 848w, https://substackcdn.com/image/fetch/$s_!dH7P!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 1272w, https://substackcdn.com/image/fetch/$s_!dH7P!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!dH7P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png" width="557" height="769.635009310987" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cc950dbb-2777-48e1-9308-490fd00bc233_537x742.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:742,&quot;width&quot;:537,&quot;resizeWidth&quot;:557,&quot;bytes&quot;:47967,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dH7P!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 424w, https://substackcdn.com/image/fetch/$s_!dH7P!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 848w, https://substackcdn.com/image/fetch/$s_!dH7P!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 1272w, https://substackcdn.com/image/fetch/$s_!dH7P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc950dbb-2777-48e1-9308-490fd00bc233_537x742.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We got the same results with matrices. But in a more clean and structured way.</p><h3><strong>WEIGHT OPTIMIZATION STRATEGY: BATCH SELECTION</strong></h3><p>We have used the first row only to optimize the random weights. Now, continue the process for all 5 rows in the dataset. After going through all rows (one full pass through the dataset), <strong>we complete one epoch. We repeat multiple epochs until the weights converge to optimal values</strong>.</p><p>For the full training dataset, we have three possible weight optimization strategies:</p><ul><li><p><strong>Stochastic Gradient Descent (SGD):</strong> Updates weights after each row (what we did so far). The term &#8216;<em>stochastic&#8217;</em> comes from probability and randomness. In Stochastic Gradient Descent (SGD), the gradient is computed using only one random sample at a time, instead of the entire dataset.</p></li><li><p><strong>Batch Gradient Descent:</strong> Updates weights after all rows in one go. Unlike Stochastic Gradient Descent (SGD) which updates weights after each row, Batch Gradient Descent updates weights only once per full dataset.</p></li><li><p><strong>Mini-Batch Gradient Descent:</strong> Uses a small batch of rows before updating weights. Mini-Batch Gradient Descent updates weights after processing a small batch of rows (e.g., 32, 64 rows at a time).</p></li></ul><h3><strong>Key Differences Between SGD and Batch Gradient Descent</strong></h3><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xo3C!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xo3C!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xo3C!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xo3C!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xo3C!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xo3C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg" width="511" height="233" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:233,&quot;width&quot;:511,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:38505,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xo3C!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xo3C!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xo3C!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xo3C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F99cc486a-28f2-40b7-b0d3-4b1ee0892c8d_511x233.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><h4><strong>Which One to Use &#8211; A Thumb Rule</strong></h4><ul><li><p><strong>SGD: </strong>If you have a huge dataset and need faster updates.</p></li><li><p><strong>Mini-Batch:</strong> If you want a balance between efficiency and stability (used in most deep learning models).</p></li><li><p><strong>Batch Gradient Descent:</strong> If you have a small dataset and can afford full-batch training.</p></li></ul><p>Whatever the weight optimization strategy is, we see that using matrix operations, especially multiplication, it makes neural networks efficient, scalable, and simple. With many neurons, computing each one separately is slow. Matrices let us calculate multiple neurons at once during forward and backward passes.</p><h3><strong>How GPUs Leverage Matrices to Accelerate Deep Learning</strong></h3><p>Now that we understand how matrices help in managing a large-scale neural network in deep learning models, let us see how GPUs take advantage of matrices properties to optimize hardware and parallel computation.</p><blockquote><p><strong>At the core, it is matrix multiplication for forward and backpropagation. If matrix multiplications can be done in parallel, that will help turbocharge neural networks and deep learning models.</strong></p></blockquote><p>GPUs can perform these multiplications in parallel, which makes training very fast. Matrix notation allows quick, vectorized calculations and helps scale to large neural networks. It also speeds up backpropagation and works efficiently with tools like <strong>TensorFlow and PyTorch.</strong></p><p>Now, let&#8217;s understand how GPUs help speed up matrix multiplication, which lies at the core of neural networks with a simple example. Again the concept goes for more complex matrices.</p><p>Let's take two simple matrices, A and B:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HH8C!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HH8C!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 424w, https://substackcdn.com/image/fetch/$s_!HH8C!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 848w, https://substackcdn.com/image/fetch/$s_!HH8C!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!HH8C!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!HH8C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg" width="422" height="429.0858208955224" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:545,&quot;width&quot;:536,&quot;resizeWidth&quot;:422,&quot;bytes&quot;:50361,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HH8C!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 424w, https://substackcdn.com/image/fetch/$s_!HH8C!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 848w, https://substackcdn.com/image/fetch/$s_!HH8C!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!HH8C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ebec59f-444f-4035-8736-bf0428e3f811_536x545.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>MATRIX MULTIPLICATION WITH GPU</strong></h3><p>A GPU typically assigns each element of the resulting matrix to a separate core (or thread) or parallel computation. For the above 2 X 2 matrix multiplication, we have 4 elements, and thus 4 parallel tasks. Each element calculation becomes its own small job:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ij3Y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ij3Y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ij3Y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ij3Y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ij3Y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!ij3Y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg" width="518" height="169.46913580246914" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:212,&quot;width&quot;:648,&quot;resizeWidth&quot;:518,&quot;bytes&quot;:40352,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ij3Y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ij3Y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ij3Y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ij3Y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbc8b565-5fb2-497b-bdf3-e383162f4a7d_648x212.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Each GPU core independently calculates one element simultaneously. This greatly reduces total computation time for large matrices.</p><h3><strong>MATRIX SPLITTING &amp; JOINING IN GPUS</strong></h3><p>In several other situations, the size of the matrices are huge and can be millions of rows/columns. In such cases, each matrix may be decomposed into smaller sizes, processed, and added back for the full output.</p><p>Here is an example of multiplication of two 4 X 4 matrices where the matrices are split, multiplied block-wise in parallel, and recombined:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xqGW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xqGW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xqGW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xqGW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xqGW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xqGW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg" width="593" height="428.12596006144395" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ace0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:470,&quot;width&quot;:651,&quot;resizeWidth&quot;:593,&quot;bytes&quot;:49495,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xqGW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xqGW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xqGW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xqGW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Face0189d-3511-4394-a82a-40d4974652e2_651x470.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Each block can be assigned to different cores, and the multiplication can be computed independently. And that is how GPUs can help with parallel processing.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!a406!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!a406!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 424w, https://substackcdn.com/image/fetch/$s_!a406!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 848w, https://substackcdn.com/image/fetch/$s_!a406!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!a406!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!a406!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg" width="515" height="115.94746716697937" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:120,&quot;width&quot;:533,&quot;resizeWidth&quot;:515,&quot;bytes&quot;:10909,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!a406!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 424w, https://substackcdn.com/image/fetch/$s_!a406!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 848w, https://substackcdn.com/image/fetch/$s_!a406!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!a406!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F047b2984-f80a-4ece-a18f-731ceadcdb55_533x120.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Here's the step-by-step calculation for C11:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CF0A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CF0A!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 424w, https://substackcdn.com/image/fetch/$s_!CF0A!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 848w, https://substackcdn.com/image/fetch/$s_!CF0A!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!CF0A!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!CF0A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg" width="621" height="463.10232558139535" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:481,&quot;width&quot;:645,&quot;resizeWidth&quot;:621,&quot;bytes&quot;:52599,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170618408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CF0A!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 424w, https://substackcdn.com/image/fetch/$s_!CF0A!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 848w, https://substackcdn.com/image/fetch/$s_!CF0A!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!CF0A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87a9e1d0-0dd4-4fb5-8131-9f081686c066_645x481.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>So, we see that subsets of a matrix can be computed independently by different GPU cores and then recombined. Matrix properties like associativity and additivity make this parallel computation possible and efficient in GPUs.</p><h3><strong>SUMMARY</strong></h3><p>Deep learning relies on large neural networks with many layers and neurons to learn complex patterns from data. These networks are trained using forward and backward propagation, where matrix multiplication is used to compute errors and update weights accordingly during the training process.</p><p>Since these operations involve large amounts of data, GPUs accelerate the process by performing matrix multiplications in parallel, leveraging the additive and associative properties of matrices to compute multiple values at once in different cores &#8212; making training faster and more efficient.</p><p>Hope this article helps to understand the fundamentals.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/why-we-need-matrices-and-gpus-for/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/why-we-need-matrices-and-gpus-for/comments"><span>Leave a comment</span></a></p><p></p><p></p>]]></content:encoded></item><item><title><![CDATA[The Real Reasons Why Companies Fail]]></title><description><![CDATA[and how to transform them fast...]]></description><link>https://www.datascienzz.com/p/the-real-reasons-why-companies-fail</link><guid isPermaLink="false">https://www.datascienzz.com/p/the-real-reasons-why-companies-fail</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Sun, 10 Aug 2025 16:36:30 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Ra-b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ra-b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ra-b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Ra-b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Ra-b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Ra-b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ra-b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg" width="847" height="347" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:347,&quot;width&quot;:847,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:69971,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170614660?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ra-b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Ra-b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Ra-b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Ra-b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5d8a377-fe8e-474b-806d-6806cb686252_847x347.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><blockquote><p>You put a frog in a bowl of water and start heating it slowly. At first, the frog enjoys the warmth. As the temperature rises, it starts adapting to the rising temperature. You turn the heat up. The frog struggles to adjust. You turn the heat up more. Now the temperature becomes unbearable. The frog finally decides to jump out. But it&#8217;s too late. It has spent all its energy adapting to the boiling water. Unable to adapt any more, it dies.</p></blockquote><p><strong>What actually killed the frog? Was it the boiling water or the frog itself for its reluctance to act when it first sensed the heat?</strong></p><div class="pullquote"><p>Several large companies die the same way.</p></div><h3><strong>WHY COMPANIES FAIL</strong></h3><p><strong>Blockbuster</strong> was once the top video rental chain, known for its stores where people rented DVDs. When the market shifted toward digital streaming, Blockbuster had an opportunity to go digital. Despite clear signs that consumer preferences were changing, the company stuck to its traditional video rental model believing its traditional rental model would stay strong forever. Reluctance to change itself allowed Netflix to take over the market &#8211; leading to Blockbuster's bankruptcy in 2010.</p><p><strong>Kodak</strong> faced a similar fate. It was a pioneer in film photography. It even invented the first digital camera in 1970s. However, the company was reluctant to fully embrace digital technology because it feared losing its profitable film business. This hesitation allowed competitors to dominate the digital market, and by the time Kodak tried to catch up, it was too late. The company eventually filed for bankruptcy in 2012.</p><p>The list goes on.</p><p><strong>Nokia</strong> was a leader in mobile phones but did not focus to shift to smartphones. It still sticked with its outdated software. Meanwhile iPhones and Android took over. By the time Nokia attempted to catch up with its partnership with Microsoft and the use of the Windows Phone operating system, it was too late. The market had already been captured by iOS and Android. Though Microsoft acquired Nokia&#8217;s phone business in 2013, but it failed and was shut down by 2016.</p><p>Other iconic brands faced similar downfall. <strong>Polaroid</strong> couldn&#8217;t transition to digital photography and filed for bankruptcy in 2001. <strong>Toys &#8220;R&#8221; Us</strong> and Borders failed to adapt to the rise of e-commerce, losing out to online giants like <strong>Amazon</strong> and ultimately going bankrupt. <strong>BlackBerry</strong>, once a dominant force in mobile phones, couldn&#8217;t keep up with the smartphone revolution, and <strong>Yahoo</strong> missed its chance to innovate, leading to its acquisition by <strong>Verizon</strong> in 2017.</p><blockquote><p><em>The examples highlight a crucial lesson: companies must transform quickly at the first sign of market shifts. Simply trying to adapt or adjust to a changing marketplace is not going to help survive.</em></p></blockquote><p><strong>So, what should be the transformation roadmap?</strong></p><p>Before we provide a roadmap let us understand how transformation process has evolved over time.</p><h3><strong>EVOLUTION OF TRANSFORMATION</strong></h3><p>Technology alone doesn't drive transformation; its impact depends on aligning with core business objectives. To understand future strategies, let's revisit key milestones in management science and corporate strategy.</p><h4><strong>1940s: FOCUS ON INDUSTRY CYCLES AND CREATIVE DESTRUCTION</strong></h4><p>In the early 20th century, businesses aimed to predict and control industry cycles, inspired by economist Joseph Schumpeter's concept of "creative destruction." This theory emphasized that industries must continually innovate to replace outdated models. For example, the automotive industry, led by companies like <strong>Ford</strong>, revolutionized production methods, setting the stage for modern manufacturing.</p><h4><strong>1950s: THE RISE OF CORPORATE STRATEGY</strong></h4><p>The 1950s marked a shift toward systematic business planning, with companies focusing on long-term corporate strategy. Firms like General Electric formalized strategic processes to set clear goals, assess market opportunities, and allocate resources efficiently.</p><p>This shift laid the groundwork for sustainable competitive advantages through structured planning.</p><h4><strong>1960s: INTRODUCTION OF SWOT ANALYSIS</strong></h4><p>The 1960s popularized <strong>SWOT analysis</strong> (Strengths, Weaknesses, Opportunities, Threats) by Albert Humphrey, providing a framework for strategic decision-making. The philosophy was to integrate internal capability S/W with external O/Ts.</p><p>Companies like IBM used this approach to assess their internal capabilities and external market threats, allowing them to adapt to technological advancements and maintain their market dominance.</p><h4><strong>1980s: COMPETITIVE ADVANTAGE AND PORTER'S THEORIES</strong></h4><p>The 1980s saw the emergence of Michael Porter's theories on competitive advantage, emphasizing <strong>"either be cheaper or be better."</strong> This era was defined by companies like Walmart, which leveraged cost leadership, and Apple, which focused on differentiation through design and innovation. Porter's frameworks, like the Five Forces Model, became essential tools for analyzing industry competition.</p><h4><strong>1990s: DISRUPTION AND THE INNOVATOR&#8217;S DILEMMA</strong></h4><p>In the 1990s, Clayton Christensen introduced the concept of "disruption" in <em>The Innovator&#8217;s Dilemma</em> (1997). His theory explains how new technologies or business models that start in niche markets can eventually disrupt established industries.</p><p>Christensen illustrates this with examples like Digital Equipment Corporation and Xerox, which drove disruptive innovation by creating agile, smaller divisions within their organizations. He also highlights how start-ups like Netflix initially targeted niche markets and later disrupted giants like Blockbuster.</p><p>This era emphasized the need for established firms to embrace disruptive innovation to remain competitive, as seen with Amazon's rise in e-commerce.</p><h4><strong>PRESENT DAY: EMBRACING DIGITAL, SUSTAINABILITY, AND ETHICAL TRANSFORMATION</strong></h4><p>Today, innovation must be fast and aligned with broader values like sustainability and ethics. Companies like Tesla are leading this charge by rapidly commercializing electric vehicles while focusing on sustainability.</p><p>Additionally, firms like OpenAI and Microsoft have leveraged AI and cloud technologies to build resilient, customer-centric operations. The current focus is on balancing rapid technological adoption with ethical considerations to ensure long-term success.</p><blockquote><p><em>These shifts emphasize the need for companies to adapt their strategies to changing market conditions by embracing technology and innovation. </em></p><p><em>The transition from traditional planning to agile, data-driven decision-making is reshaping the business landscape.</em></p></blockquote><p>Now let us outline the transformation roadmap.</p><h3><strong>TRANSFORMATION ROADMAP IN CURRENT DIGITAL ERA</strong></h3><p>Following the above trend on the transformation strategies in the current digital era, below we outline a roadmap for a successful digital transformation.</p><h4><strong>DEFINE A CLEAR STRATEGY</strong></h4><p>Before starting a digital transformation, evaluate where your business stands in terms of technology, processes, and customer experiences to identify areas for improvement. Once you have a clear understanding, define specific outcomes you want to achieve, such as enhancing customer experience, improving operational efficiency, expanding to new markets, or driving innovation. Setting clear goals will help guide your transformation strategy.</p><p><strong>Netflix</strong> successfully transformed from a DVD rental service to a global streaming platform by embracing digital technology and using data to tailor content to viewer preferences. Similarly, <strong>Amazon</strong> evolved from an online bookstore into the world&#8217;s largest e-commerce platform, leveraging cloud computing, big data, and AI to optimize operations and innovate across industries.</p><h4><strong>FOCUS ON CUSTOMER-CENTRICITY</strong></h4><p>A customer-centric approach is essential for successful digital transformation. Businesses must understand their customers&#8217; needs and pain points, focusing on delivering personalized experiences, seamless interactions, and innovative products or services.</p><p>For example, <strong>Sephora</strong> uses customer-centric tools like its "Virtual Artist" app, which lets users try on makeup virtually using augmented reality. Sephora also uses influencer marketing through which shoppers can visualize the products' use by people they follow and trust.</p><p>This goes a long way enhancing the shopping experience and driving sales.</p><h4><strong>TECHNOLOGY &amp; INNOVATION</strong></h4><p>Digital transformation is not just about adopting new technology; it&#8217;s about reshaping your company&#8217;s mindset and processes. Businesses should leverage emerging technologies like AI, machine learning, cloud computing, IoT, and data analytics to optimize operations, enhance decision-making, and improve customer experiences.</p><blockquote><p><em>It is important that technology should follow strategy. </em></p><p><em>However, many companies do it the reverse. </em></p><p><em>They select a technology and then let the business strategy follow that. </em></p></blockquote><div class="pullquote"><p><em>The result is miserable.</em></p></div><p>So have a great strategy and select the technology that supports your strategy.</p><p>One example of a company that successfully aligned technology with its strategy is <strong>Apple</strong>. It successfully aligns its technology with its strategy by focusing on seamless, user-friendly experiences through high-quality hardware and software integration, creating a cohesive ecosystem that supports its customer-centric vision and drives brand loyalty.</p><p>Again, technology alone is not enough. It must be backed by a culture of innovation and adoption. We cannot undermine the importance of cultivating a culture that embraces change, encourages experimentation, and promotes continuous learning.</p><h4><strong>PROCESS OPTIMIZATION &amp; AUTOMATION</strong></h4><p>Process optimization and automation play a key role in digital transformation by streamlining workflows, reducing manual tasks, and increasing operational efficiency. By optimizing and automating repetitive processes, businesses can minimize errors, save time, and allocate resources more effectively. </p><p>This allows companies to focus on more strategic activities, improve decision-making with real-time data, and enhance customer experiences.</p><p>Ultimately, process optimization and automation drive cost savings, productivity, and innovation, helping businesses adapt quickly to changing markets and stay competitive.</p><p>A great example is <strong>General Electric (GE)</strong>, which implemented automation in its manufacturing processes, especially in its aviation division. GE uses predictive maintenance technology to monitor equipment health, automating the process of identifying potential issues before they cause downtime. </p><p>This not only improves operational efficiency but also reduces maintenance costs and enhances product reliability, supporting GE's digital transformation efforts.</p><h4><strong>COLLABORATION</strong></h4><p>Collaboration is crucial in digital transformation as it brings together diverse teams, ideas, and expertise to drive innovation and efficiency. </p><p>By fostering cross-functional collaboration between departments such as IT, marketing, operations, and customer service, businesses can create more integrated solutions that align with both customer needs and business goals. </p><p>Collaboration also encourages knowledge sharing, faster problem-solving, and the ability to quickly adapt to new technologies or market changes.</p><p>For example, <strong>Microsoft</strong> successfully shifted from a software company to a cloud services provider by fostering collaboration across teams and partnering with other tech providers, leading to its success in cloud computing. </p><p>Similarly, <strong>Toyota&#8217;s</strong> collaboration between engineering, design, and technology teams, along with partnerships with tech firms, has helped the company stay at the forefront of automotive innovation, especially with hybrid and electric vehicles.</p><p>Without collaboration, any transformation initiative is likely to face internal resistance and will struggle to succeed.</p><h4><strong>LEADERSHIP</strong></h4><p>Successful transformation depends on support from all levels of the organization, especially leadership. Leaders must be committed and aligned with the transformation vision.</p><blockquote><p>"<em>Charity begins at home</em>." </p><p>To lead successful transformation, leaders must first transform themselves.</p></blockquote><p>They should act as catalysts, creating the right environment for change by fostering collaboration, innovation, and curiosity.</p><p>In an ever-evolving world, leaders need to adopt a mindset of exploration&#8212;embracing curiosity to see both the big picture (telescopic) and the details (microscopic). They need to remain in <em>present</em> while focusing on the <em>future</em>.</p><p>Leadership requires courage to experiment, engage with their teams, and communicate authentically. Trust and a commitment to living core values are essential, even in the face of resistance. </p><p>Leaders who embrace emotional intelligence, curiosity, and adaptability can guide their organizations through change with conviction and inspire their teams to innovate and thrive in a rapidly evolving world.</p><h4><strong>ITERATE &amp; SCALE</strong></h4><p>Digital transformation is not a one-time event; it&#8217;s an ongoing journey of continuous improvement. Leaders suggest starting with small, manageable pilot projects that allow the organization to experiment, test, and learn from real-world results.</p><p>These initial projects provide valuable insights and help refine strategies. Once a pilot initiative proves successful, it can be scaled across the organization, ensuring broader adoption and impact.</p><p>This iterative approach minimizes risk, fosters agility, and enables leaders to fine-tune their strategies before full-scale implementation, ultimately driving sustainable transformation across the business.</p><p><strong>Spotify</strong> and <strong>Starbucks</strong> provide strong examples of iterating and scaling during digital transformation. <strong>Spotify</strong> began with a small, localized pilot of its music streaming service, gathering feedback to refine features before expanding globally.</p><p>Similarly, <strong>Starbucks</strong> launched a mobile app with basic ordering functions, then enhanced it based on customer feedback by adding features like loyalty rewards and personalized offers.</p><p>Both companies used small-scale pilots to test, refine, and optimize their digital strategies before scaling them across their businesses, ensuring greater success and customer engagement.</p><h4><strong>DATA ANALYTICS &amp; AI/ML</strong></h4><p>A data-driven approach is essential for digital transformation, enabling businesses to make informed decisions, optimize operations, and enhance customer experiences.</p><p>Several success stories of diverse organizations emphasize the importance of using data, along with advanced technologies like AI/ML, to drive efficiency and personalization.</p><p>For example, <strong>Walmart</strong> uses data analytics powered by AI/ML to manage inventory, predict demand, and optimize supply chains, ensuring products are always in stock.</p><p><strong>Uber</strong> leverages real-time data and machine learning algorithms to improve ride matching, optimize routes, and adjust pricing dynamically based on demand.</p><p>By integrating AI/ML with data, companies can enhance operations, provide personalized services, and stay ahead in a rapidly evolving market.</p><h3><strong>SUMMARY</strong></h3><p>In the rapidly evolving digital era, relying on gradual adaptation is much like the frog in the boiling water. </p><p>Neither does Lamarck's theory of adaptation. </p><p>Companies must act swiftly and strategically to stay ahead.</p><blockquote><p><em>As Mark Twain once said, &#8220;If you tell the truth, you don&#8217;t need to remember anything.&#8221;</em></p><p>Similarly, if you have a clear transformation strategy and vision, communicate it effectively to all stakeholders, and everything else will fall into place.</p></blockquote><p>Businesses must develop a digital transformation strategy that is not only forward-thinking but also practical and aligned with their long-term goals. </p><p>Such a robust strategy serves as the guiding principle for every aspect of transformation&#8212;whether it's collaborating with stakeholders, selecting the right technologies, or optimizing business processes. </p><p>By staying committed to a well-defined strategy, companies can successfully navigate the complexities of digital change and achieve sustainable success.</p><p>Let me know what you think.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/the-real-reasons-why-companies-fail/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/the-real-reasons-why-companies-fail/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Why Quantum Matters Now]]></title><description><![CDATA[All about Quantum &#8211; Hype, Reality, and the Road Ahead]]></description><link>https://www.datascienzz.com/p/why-quantum-matters-now</link><guid isPermaLink="false">https://www.datascienzz.com/p/why-quantum-matters-now</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Sun, 10 Aug 2025 16:08:06 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!wSuF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!wSuF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wSuF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!wSuF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!wSuF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!wSuF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!wSuF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png" width="1024" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:608,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!wSuF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!wSuF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!wSuF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!wSuF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbde9faa-fb4a-443c-abe4-16e44dc06e34_1024x608.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">quantum computation</figcaption></figure></div><p><strong>Recently, Google has developed a quantum chip which it <a href="https://www.bbc.com/news/articles/c791ng0zvl3o">claims</a> to solve a problem in just 5 minutes that would take a classical supercomputer 10 septillion or 10<sup>25</sup> years to complete.</strong></p><p>Let&#8217;s understand the statement deeply &#8211; especially how long is 10 septillion years exactly.</p><blockquote><p>The age of earth is 4.5 billion years (4.5 &#215; 10&#8313;) and the age of our universe is 13.8 billion years (1.38 &#215; 10&#185;&#8304;). So, 10 septillion years is 7 &#215; 10<sup>15</sup> times longer than the age of the universe. </p></blockquote><p><strong>That means, if the entire age of our universe were squeezed into 1 second, then 10 septillion years would be 23 million years.</strong></p><div class="pullquote"><p>Simply mind-blowing!</p></div><blockquote><h3><strong>Why quantum computation is so fast?</strong></h3></blockquote><p>It&#8217;s because of their fundamental designs.</p><p>A Classical computer uses binary bits 0 or 1 &#8211; only one at a time. But Quantum computers use qubits, which can be 0/1 or both at the same time &#8211; due to a quantum property called superposition.</p><p>This means classical computers process one state at a time, while quantum computers can process many states simultaneously.</p><p>To understand better, look at the below table:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!d8-F!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!d8-F!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 424w, https://substackcdn.com/image/fetch/$s_!d8-F!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 848w, https://substackcdn.com/image/fetch/$s_!d8-F!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!d8-F!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!d8-F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg" width="700" height="472" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:472,&quot;width&quot;:700,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:102881,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170612319?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!d8-F!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 424w, https://substackcdn.com/image/fetch/$s_!d8-F!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 848w, https://substackcdn.com/image/fetch/$s_!d8-F!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!d8-F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F374e2a7e-76dc-4ad0-8d7b-cc8fbf3acd33_700x472.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That&#8217;s how quantum computing can be exponentially faster compared to any traditional computer.</p><h3><strong>The fundamental Theory</strong></h3><p>Quantum computers harness the unique rules of quantum mechanics to process information in a radically different way. Instead of bits, they use qubits &#8211; tiny quantum systems like electrons or photons that behave as both particles and waves.</p><p>Because of their wave-like nature, qubits can exist in a superposition &#8211; meaning they can be in multiple states (0 and 1) at the same time, not just one or the other like classical bits. This allows quantum computers to explore many possible solutions simultaneously.</p><p>To keep this delicate quantum behavior stable, qubits are kept at ultra-cold temperatures, <strong>close to absolute zero (&#8722;273.15&#176;C).</strong> At this point, thermal noise is minimal, and the quantum states are less likely to break down (a problem known as <em>decoherence</em>).</p><p>Quantum engineers then use precise energy pulses &#8211; such as microwaves or lasers to carefully control the qubits. By doing so, they perform calculations through a combination of <em>superposition</em>, <em>entanglement</em>, and <em>quantum interference</em>, allowing for powerful parallel computation that classical computers can&#8217;t match.</p><h3><strong>Why Businesses Need Quantum Computing</strong></h3><p>Quantum computing isn&#8217;t just faster &#8211; it can handle problems that are impossible for classical computers due to massive size and complexity. Here's how it's set to transform industries:</p><h4><strong>Breaking Through Classical Limits</strong></h4><p><strong>The End of Moore&#8217;s Law:</strong> Transistors in classical chips are approaching atomic sizes, slowing down decades of progress. We're nearing the physical limits of classical scalability.</p><h4><strong>Large Scale Optimization Problems</strong></h4><p>Classical computers struggle with problems that grow exponentially &#8212; like optimizing global supply chains with billions of possible routes. Quantum systems can handle these far more efficiently.</p><ul><li><p><strong>Traveling Salesperson Problem (TSP):</strong> Finding the shortest route between 2,000 cities requires evaluating over 10^5,000 possibilities. Classical supercomputers cannot brute-force this in any practical time. Quantum algorithms can theoretically evaluate multiple routes simultaneously, making such problems feasible.</p></li><li><p><strong>Finance:</strong> Real-time portfolio optimization, derivative pricing, and risk analysis are extremely complex and requires intensive computation. Quantum methods can provide near-instant insights.</p></li></ul><h4><strong>Energy Efficiency</strong></h4><p>According to the <strong>International Energy Agency (IEA)</strong>, data centres globally consume 1&#8211;2% of the world's electricity with a potential <strong>doubling by 2026</strong> due to increasing demands from AI and cloud computing services. Quantum systems, by solving certain problems in fewer steps, could be vastly more energy-efficient for specific workloads.</p><h4><strong>Quantum Random Numbers</strong></h4><p>Classical computers, uses deterministic algorithms like Linear Congruential Generator (LCG) to generate pseudo-random numbers. However, quantum systems can produce truly random numbers using quantum algorithms &#8211; making cryptographic systems much more secure and resistant to prediction or attacks.</p><h4><strong>Searching Unstructured Data</strong></h4><p>Imagine you have a haystack with 1 million wooden needles, but only one is made of iron. With a classical computer, you'd pick up and check one needle one by one. In the worst case, you go through all 1 million to find the iron needle. So, the complexity is O(N).</p><p>Now imagine using a quantum magnet that pulls any iron needle. You wave it over the haystack, and instead of checking every needle, you find the iron one in just about 1,000 tries &#8211; the square root of 1 million. That&#8217;s O(&#8730;N).</p><p>For small problems, the speedup isn&#8217;t huge. But for huge databases &#8211; say, searching through 1 trillion entries &#8211; Grover&#8217;s Algorithm can reduce search time from 1 trillion steps to just 1 million.</p><p>Such algorithms can help in:</p><blockquote><ul><li><p><strong>AI &amp; Machine Learning:</strong> Quickly find optimal hyperparameters or best matches in large datasets.</p></li><li><p><strong>Search Engines:</strong> Improve keyword or document matching in huge databases.</p></li><li><p><strong>Fraud Detection:</strong> Rapidly identify suspicious patterns in financial transactions.</p></li></ul></blockquote><h4><strong>Accelerating AI &amp; Machine Learning</strong></h4><p>Classical deep learning models can take days or weeks or months to train on large datasets. Quantum systems could cut this down drastically with quantum-enhanced linear algebra.</p><p>Some areas that can benefit from are:</p><blockquote><ul><li><p><strong>Fraud detection</strong> across millions of transactions in real-time.</p></li><li><p><strong>Facial recognition</strong> in smart cities.</p></li><li><p><strong>Recommendation systems</strong> that can adapt faster to user behavior.</p></li><li><p><strong>LLM, AI Agent, and AGI</strong> Models</p></li></ul></blockquote><h4><strong>Driving Scientific Discovery</strong></h4><p>Few notable areas are:</p><ul><li><p><strong>Drug Discovery:</strong> Simulating a single molecule like penicillin involves modeling quantum interactions between thousands of electrons &#8212; classically intractable. Quantum computers can simulate molecular behavior natively, potentially cutting years off drug development cycles.</p></li><li><p><strong>DNA Synthesis &amp; Protein Folding:</strong> Simulating how a long DNA or protein strand folds (crucial in drug design) can take months or years of supercomputing time. Quantum computers can model these interactions exponentially faster, accelerating biotech R&amp;D.</p></li><li><p><strong>Climate Modeling:</strong> Predicting climate systems involves trillions of variables &#8212; quantum methods can improve accuracy and speed, helping us respond to environmental challenges.</p></li><li><p><strong>Green Chemistry:</strong> Quantum systems could design energy-efficient catalysts, replacing energy-intensive processes like the Haber-Bosch method, which consumes 1&#8211;2% of global energy to make fertilizer.</p></li></ul><h3><strong>Challenges and Threats in Quantum Computing</strong></h3><p>While quantum computing holds enormous potential, it also faces several serious technical, economic, and security-related hurdles that must be addressed before widespread adoption.</p><h4><strong>Technical Immaturity</strong></h4><p>Quantum computers today are still highly error-prone. The delicate state of a qubit can collapse due to noise, vibrations, or even slight temperature fluctuations &#8212; a phenomenon known as decoherence. To build a single stable logical qubit, it may take thousands of physical qubits working together.</p><p>As of 2024, IBM&#8217;s most advanced system, Condor, has 1,121 qubits, but fully error-corrected systems will likely need millions.</p><h4><strong>High Infrastructure Costs</strong></h4><p>Quantum systems require extremely controlled environments, including cryogenic <strong>temperatures near absolute zero (&#8722;273&#176;C),</strong> vibration isolation, electromagnetic shielding, and precision control hardware. Such requirements make quantum systems expensive to build and maintain.</p><p>IBM, Google, and Honeywell have collectively invested billions of dollars into quantum research and infrastructure. Only a handful of tech giants and research institutions can currently afford this.</p><h4><strong>Cybersecurity Risks</strong></h4><p>Quantum computing poses a direct threat to current cryptographic systems. Algorithms like RSA, ECC, and DSA are widely used today for secure communication &#8211; from banking transactions to government defense systems. These algorithms rely on mathematical problems (e.g., factoring large integers) that are hard for classical computers but easy for quantum machines using Shor&#8217;s Algorithm.</p><p>A powerful enough quantum computer could break 2048-bit RSA encryption in minutes to hours, exposing financial data, healthcare records, government secrets, intellectual property stored in cloud.</p><blockquote><p><strong>There's also a risk of &#8216;store now, decrypt later&#8217; attacks.</strong> It means hackers are collecting encrypted data today, even if they can&#8217;t read it yet. They plan to decrypt it in the future using powerful quantum computers. This is a big risk for cyber-security once quantum tech is ready.</p><p>That&#8217;s why, in 2022, the U.S. government passed the Quantum Computing Cybersecurity Preparedness Act, pushing federal agencies to begin transitioning to quantum-safe encryption.</p></blockquote><h4><strong>Talent Shortage</strong></h4><p>Quantum computing sits at the intersection of physics, math, engineering, and computer science. However, there are fewer quantum scientists and engineers globally with practical experience. A massive skills gap exists, with universities only recently introducing structured quantum programs. Until the workforce catches up, progress will remain limited to a small elite.</p><h4><strong>Ethical and Geopolitical Concerns</strong></h4><p>Access to quantum technologies is likely to be uneven, raising concerns about: </p><blockquote><ul><li><p><strong>Technological Monopolies:</strong> Corporations or countries with early access could dominate sectors like finance, defence, and pharmaceuticals.</p></li><li><p><strong>Global Inequality:</strong> Nations without the infrastructure or investment capital could be left behind, further widening digital divides.</p></li><li><p><strong>Military Misuse:</strong> Quantum capabilities like <strong>cryptanalysis (breaking encryption)</strong> can spark a new form of tech arms race. A country with a large quantum computer could break into encrypted military communications of other nations.</p></li><li><p>Similarly, <strong>quantum sensors</strong> can detect submarines, stealth aircraft, or underground bunkers with extreme precision, even when traditional radar or sonar fails.</p></li></ul></blockquote><p>Access to quantum technologies is likely to be uneven, raising concerns about: </p><p>A nation with advanced quantum sensing could gain a huge intelligence edge, making current stealth technology obsolete.</p><p>Such kind of tech edge could shift global power dynamics &#8211; prompting nations to race to develop and control quantum capabilities, just like it happened with nuclear and AI.</p><h4><strong>Investment Timing Challenge</strong></h4><p>Quantum computing is promising &#8212; but not quite enterprise-ready. Most companies have already sunk serious investment into traditional AI including, cloud, GPUs, ML pipelines and they're seeing real returns now. Switching to quantum means big spending on new hardware and talent, with unclear short-term payoff.</p><p>For now, the smarter move is to maximize existing AI while keeping a close eye on quantum&#8217;s evolution.</p><h3><strong>Quantum Adoption Strategy</strong></h3><p>Jumping directly into quantum hardware isn't practical for most businesses today. Instead, a more realistic path is a hybrid approach &#8212; using classical systems alongside quantum-inspired techniques. This lets organizations benefit from quantum thinking without needing to invest heavily in complex infrastructure right away.</p><p>We&#8217;ve seen similar transitions before. Hybrid cars, for example, paved the way for fully electric vehicles.</p><p>Likewise, industries moved gradually from analog to digital systems to avoid disruption. Today, several companies like HPE, Mphasis, Infosys are applying quantum-inspired algorithms on classical machines to solve problems in areas like optimization and AI &#8211; gaining immediate value while preparing for future quantum integration.</p><h3><strong>When Will Quantum Computing Go Mainstream?</strong></h3><p>Quantum computing is still in its early stages but progressing steadily &#8212; much like classical computing did:</p><blockquote><ul><li><p>Transistor invented: 1947</p></li><li><p>Commercial CPUs emerge: 1970s (e.g., Intel 4004)</p></li><li><p>Widespread adoption: 1990s&#8211;2000s</p></li><li><p>Similarly, early development started around 2020s, and adoption is expected after a decade or two &#8211; who knows!</p></li></ul></blockquote><p>Big tech enterprises are investing in quantum now. Google aims to build a useful quantum computer by 2029. Microsoft Azure, Amazon, and IBM already offer cloud-based quantum access for developers and researchers.</p><h3><strong>Embracing the Quantum Future</strong></h3><p>Quantum computing is a game-changer &#8211; solving problems once thought impossible, from optimization and AI to cybersecurity and scientific breakthroughs.</p><p>However, the rise of quantum computing is a double-edged sword &#8211; it promises transformative power but also introduces unprecedented risks. Managing this transition will require global cooperation, forward-thinking policy, and investment in secure infrastructure.</p><p>Quantum adoption isn&#8217;t just plug-and-play. Businesses need a strategic, hybrid approach while the tech matures. The quantum era isn&#8217;t just a tech shift &#8211; it&#8217;s a competitive edge. Those who prepare now will lead tomorrow.</p><p>For most businesses, now is the time to explore quantum-inspired algorithms, build internal awareness, and develop long-term strategies &#8211; while quantum hardware continues to evolve.</p><div class="pullquote"><p>The final question is: What&#8217;s your quantum strategy?</p></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/why-quantum-matters-now/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/why-quantum-matters-now/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Why Less is More in ML]]></title><description><![CDATA[A Simple Illustration of L0, L1, and L2 Regularizations]]></description><link>https://www.datascienzz.com/p/why-less-is-more-in-ml</link><guid isPermaLink="false">https://www.datascienzz.com/p/why-less-is-more-in-ml</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Sun, 10 Aug 2025 14:16:19 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!iOUa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!iOUa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!iOUa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 424w, https://substackcdn.com/image/fetch/$s_!iOUa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 848w, https://substackcdn.com/image/fetch/$s_!iOUa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!iOUa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!iOUa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg" width="728" height="355.264" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:366,&quot;width&quot;:750,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:36058,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170603678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!iOUa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 424w, https://substackcdn.com/image/fetch/$s_!iOUa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 848w, https://substackcdn.com/image/fetch/$s_!iOUa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!iOUa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc63d526-9b01-41d8-a8bd-9794e03c70f7_750x366.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="pullquote"><p>The goal of machine learning models is not to perfectly fit the training data &#8211; it&#8217;s to make accurate predictions on new, unseen data.</p></div><p>More data or bigger models don&#8217;t always lead to better results in machine learning. In fact, complex models often results in overfitting &#8212; where a model fits the training data very well but struggles with unseen data.</p><h3><strong>Why Simple Models are Better &#8211; Occam&#8217;s Razor Principle</strong></h3><p>A simpler model is usually more robust, easier to interpret, faster to train, and less likely to overfit. According to <strong>Occam&#8217;s Razor principle</strong>, simpler models are generally better than more complex ones.</p><blockquote><p><em>For example, when predicting house prices, we could build a complex model that considers every detail of the house like door colour, garden flowers, mailbox design, and so on. But we can keep it simple just by using key factors like number of bedrooms, square foot area, and house location.</em></p><p><em>Same goes for predicting student scores. A complex model might track pen colour, shoe brand, desk angle. But a simple model using hours studied and past performance is more practical &#8212; and often just as accurate.</em></p></blockquote><p>The idea is to keep it simple because a simpler model is often the best choice!</p><h3><strong>So, How Simple Should a Model Be?</strong></h3><p>A simple model is usually better but often underfits. Because, it is too simple, it misses patterns. <strong>We call it as &#8216;Bias Error&#8217;.</strong></p><p>On the other side, a complex model often overfits the data by memorizing noise. Though it captures pattern on training data does not fit on unseen, new data. <strong>We call it as &#8216;Variance Error&#8217;.</strong></p><p>An optimal model should be a balanced one.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ziHt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ziHt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ziHt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ziHt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ziHt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!ziHt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg" width="611" height="112" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:112,&quot;width&quot;:611,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:30962,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170603678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ziHt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ziHt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ziHt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ziHt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd5d74c6-712a-4e25-9931-8a4cc9537e1f_611x112.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Let us understand visually.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://substackcdn.com/image/fetch/$s_!S0-s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S0-s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 424w, https://substackcdn.com/image/fetch/$s_!S0-s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 848w, https://substackcdn.com/image/fetch/$s_!S0-s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!S0-s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!S0-s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg" width="572" height="230" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:230,&quot;width&quot;:572,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:19788,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170603678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!S0-s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 424w, https://substackcdn.com/image/fetch/$s_!S0-s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 848w, https://substackcdn.com/image/fetch/$s_!S0-s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!S0-s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ce35583-1346-462f-93c4-a7807416127f_572x230.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>Imagine, you are trying to hit the target with arrows.</p><ul><li><p>In underfitting (high bias), attempted arrows are mostly consistent but away from the target &#8212; showing that the model needs some more info to capture the pattern.</p></li><li><p>In overfitting (high variance), arrows are scattered all over including the target. The model captures the pattern, but the performance is not stable.</p></li><li><p>A balanced model generalizes well and is optimal level of bias and variance.</p></li></ul><h3><strong>How to Keep ML Models Under Check?</strong></h3><div class="pullquote"><p>It&#8217;s Regularization.</p></div><p>Regularization helps to keep your ML model on track by shrinking, removing, or simplifying model parameters (weights). The objective of regularization is to make model simpler &#8211; but not the simplest. Simpler models with regularization often make better predictions.</p><p>Now we&#8217;ll discuss how techniques like L0, L1, and L2 regularization improve generalization, reduce errors, and lead to more efficient models.</p><h3><strong>What is Regularization?</strong></h3><p>Regularization is a way to keep your model from getting too complicated.</p><blockquote><p>Think of it as telling your model, &#8220;Don&#8217;t overthink it.&#8221;</p><p>It works by adding a penalty to the loss function, which discourages the model from getting too complex.</p></blockquote><h4><strong>Base Loss Function: MSE</strong></h4><p>Let&#8217;s start with the standard Mean Squared Error (MSE) loss:</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\mathrm{MSE} = \\frac{1}{2n} \\sum_{i=1}^n \\left( y_i - \\hat{y_i} \\right)^2 &quot;,&quot;id&quot;:&quot;FTSJUODQZC&quot;}" data-component-name="LatexBlockToDOM"></div><p>We minimize this to train models. But if we add regularization, we modify the loss function as below:</p><p>Loss = MSE + &#120582; Penalty                                                                                           &#8230; &#8230; (1)</p><p>Here, &#120582; controls how much penalty to apply. And the penalty depends on the type of regularization.</p><h3><strong>The Three Types of Regularization</strong></h3><p>There are Three regularization techniques L0, L1, and L2. Let&#8217;s understand them in simple terms.</p><h4><strong>L0 Regularization &#8211; &#8216;Pick the Fewest&#8217;</strong></h4><p>This is the ideal regularization technique that wants to add the minimum number of features to use in a model and hence, adds penalty to the number of non-zero weights.</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\text{Loss} = \\text{MSE} + \\lambda \\lVert w \\rVert_{0}\n\n\n\\quad \\text{Or,} \\quad\n\\text{Loss} = \\frac{1}{2n} \\sum_{i=1}^n (y_i - \\hat{y}_i)^2 + \\lambda \\lVert w \\rVert_{0}\n&quot;,&quot;id&quot;:&quot;YJIXOXRLMF&quot;}" data-component-name="LatexBlockToDOM"></div><p>The term ||w||o defines the number of non-zero weights in a weight vector. For example, </p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\text{if,} \\quad w = [4.5, 2, 0, -1, 0] &quot;,&quot;id&quot;:&quot;KCHUNUDBEN&quot;}" data-component-name="LatexBlockToDOM"></div><p>then  </p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\\\[6pt]\n\\lVert w \\rVert_{0} = 3 \\quad \\text{(because 3 weights are non-zero)}\n&quot;,&quot;id&quot;:&quot;DMKIJAVYFP&quot;}" data-component-name="LatexBlockToDOM"></div><p>So, L0 regularization pushes the model to use as few features as possible. It&#8217;s like packing for a trip with strict baggage limits&#8212;you only bring what&#8217;s absolutely essential. And hence, most weights drop to exactly zero and results in a minimal, highly sparse model.</p><h4><strong>L1 Regularization (Lasso) &#8211; &#8216;Drop Some&#8217;</strong></h4><p>Here the penalty factor is the sum of absolute weights:</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\text{Loss} = \\text{MSE} + \\lambda \\sum |w_i|  \n\n\\quad \\text{Or,} \\quad\n\\text{Loss} = \\frac{1}{2n} \\sum_{i=1}^n (y_i - \\hat{y}_i)^2 + \\lambda \\sum |w_i|\n&quot;,&quot;id&quot;:&quot;GFMHBYKPSA&quot;}" data-component-name="LatexBlockToDOM"></div><p>So, L1 regularization pushes some weights drop to zero. It&#8217;s like you&#8217;ve got limited luggage space, so you pack only what you&#8217;ll actually use. With this, it automatically selects the most important features that truly matter. Hence helps in feature selection also.</p><h4><strong>L2 Regularization (Ridge) &#8211; &#8216;Shrink All&#8217;</strong></h4><p>Here the penalty factor is the sum of squared weights:</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;\\text{Loss} = \\text{MSE} + \\lambda \\sum w_i^2  \n\n\\quad \\text{Or,} \\quad\n\\text{Loss} = \\frac{1}{2n} \\sum_{i=1}^n (y_i - \\hat{y}_i)^2 + \\lambda \\sum w_i^2\n&quot;,&quot;id&quot;:&quot;BGRGZTGDOT&quot;}" data-component-name="LatexBlockToDOM"></div><p>L2 regularization shrinks all weights without setting them to zero. It&#8217;s like, you can pack everything, but it all has to fit in a carry-on. This is used when every feature adds value, but none should dominate.</p><h3><strong>How is the Penalty Function Optimized during Training?</strong></h3><p>L0 regularization decides which features should stay and which should go. It&#8217;s a combinatorial optimization problem. Since it&#8217;s not differentiable or convex, standard optimization methods like gradient descent won&#8217;t work. Instead, it relies on greedy algorithms, heuristics, or brute force &#8212; similar to traditional feature selection.</p><p>So, though L0 provides the best regularization model, it is computationally expensive, hardest to optimize, and difficult to execute.</p><h4><strong>Hence, we mostly use L1 and L2.</strong></h4><blockquote><ul><li><p><strong>L1 regularization is convex</strong> <strong>but</strong> <strong>not smooth</strong>, making it moderately easy to optimize with specialized methods; it promotes sparsity by setting some weights to zero.</p></li><li><p><strong>L2 regularization is both convex and smooth,</strong> making it the easiest to optimize using standard methods, and it shrinks weights without eliminating them.</p></li></ul></blockquote><p>A simple summary is provided below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hOD8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hOD8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hOD8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hOD8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hOD8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!hOD8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg" width="775" height="282" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:282,&quot;width&quot;:775,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:65661,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://datascienzz.substack.com/i/170603678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hOD8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hOD8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hOD8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hOD8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b837546-e55e-4ced-a070-29b2c263c641_775x282.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>SUMMARY</strong></h3><ul><li><p>Regularization helps ML models stay simple, avoid overfitting, and generalize better.</p></li><li><p>By shrinking, dropping, or limiting weights using L0, L1, or L2 methods, we find the right balance between bias and variance.</p></li><li><p>In ML, simpler models often perform better and that&#8217;s the power of &#8216;less is more&#8217;.</p></li></ul><blockquote><p><strong>Simplicity is a strategy. Keep it simple. Make it smart.</strong></p></blockquote><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/why-less-is-more-in-ml/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/why-less-is-more-in-ml/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[The Cobra Effect in Data Science]]></title><description><![CDATA[A good intention to solve a problem is not just enough...]]></description><link>https://www.datascienzz.com/p/the-cobra-effect-in-data-science</link><guid isPermaLink="false">https://www.datascienzz.com/p/the-cobra-effect-in-data-science</guid><dc:creator><![CDATA[Santanu Sinha]]></dc:creator><pubDate>Sun, 10 Aug 2025 12:49:52 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!R2gf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!R2gf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!R2gf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!R2gf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!R2gf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!R2gf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!R2gf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png" width="1024" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:608,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!R2gf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 424w, https://substackcdn.com/image/fetch/$s_!R2gf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 848w, https://substackcdn.com/image/fetch/$s_!R2gf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 1272w, https://substackcdn.com/image/fetch/$s_!R2gf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff172d9dc-dba8-4d20-8738-adaf1b554aef_1024x608.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p>Imagine, you have a problem. You also have found a solution. The solution initially works very well. However, after some time, the solution &#8211; instead of solving the problem &#8211; makes the problem even worse. </p><p><strong>This is the 'Cobra effect' &#8211; an interesting concept from political economics.</strong></p><blockquote><p>The term came from the story of a policy issued by the British government in India.</p><p>Once, there was a huge population of venomous cobras in a part of the country. The government declared a silver coin as a reward for each dead cobra. This motivated people to proactively hunt cobras and eventually get rewarded.</p><p>Everything was going fine until some smart people found it more profitable to breed cobras in-house and later convert them into silver coins. </p><p>When this news came out, the government scrapped the policy. The cobra breeders had to set the cobras free, as it was no longer a lucrative trade option. </p><p>Eventually, with the released cobras, the cobra problem became even worse!</p></blockquote><div class="pullquote"><p>Just to interpret the story, sometimes a good intention to curb a problem &#8211; does not necessarily end with a desirable result.</p></div><p>In today's world, we have a lot of digital data, and organizations are using it to generate insights and solve business problems. This created a supersonic era of Data Science. Data Science is a buzzword. However, sometimes the results are not what we intend.</p><p>Recently, I have talked to several industry practitioners, thought leaders, and researchers about this issue, and they've shared some examples of the Cobra effect from data science. Let's take a look at a few of them.</p><h3><strong>Data models built in silos</strong></h3><p>Consider a large organization. It has several product divisions and business functions. Often, they work independently and in silos. Data models of a division often do not talk to the data models of others. As a result, the outcomes are often sub-optimal and counter-intuitive.</p><p>For example, the analytics manager of procurement advises buying in bulk to leverage the economies and higher discounts from the suppliers. However, the benefits are often negated by inventory holding costs, opportunity costs, and the risk of the products being obsolete shortly. </p><p>Similarly, an inventory manager asks her data science team to come up with a great inventory optimization model that thrives to save on inventory holding costs. Though, the model can save some dollars on the inventory &#8211; it may fail to fulfil a million-dollar order due to material shortages &#8211; if the model does not take inputs from the sales and marketing team!</p><p>Such examples show how the original intention of a business division can be great, but without a holistic view of the overall business landscape, it may fail to bring in the desired benefits. It's important for organizations to break down data silos and ensure that data models talk to each other, to achieve the best outcomes for the entire business.</p><h3><strong>Too many dashboards &amp; reports</strong></h3><p>In an organization, there are usually several data science or analytics teams. Now each analytics team builds sophisticated visualization and business reports. These are discussed in the board room. Often, the data source and model assumptions are different, and hence the reports capture only partial, incomplete, and conflicting views of the business. Hence the executives are left dark and clueless.</p><p>To please the executives, more reports are produced again. And the process keeps on going across teams. Over some time, there is a flood of reports without a consensus. It takes a significant amount of time and effort to build those reports &#8211; which ultimately gets wasted.</p><p>It's important for organizations to ensure that all teams are working with the same data sources and assumptions so that the reports generated give a complete and accurate view of the business. This way, executives can make informed decisions based on the data, without getting lost in a sea of conflicting reports.</p><h3><strong>Unexplainable predictive models</strong></h3><p>The data science team builds sophisticated predictive models to solve a business problem. The analysts spend long hours collecting, cleaning, processing, and modelling the data. Sometimes, the models are too complex and turn out to be an unexplainable black box.</p><p>On top of that, during the development, often there is not much involvement of the domain experts. As a result, the models are off the ground &#8211; without proper business alignment or foundation. Such models look okay basis the given training data. However, when deployed in production, the models fail badly and the result is disastrous.</p><p>The other day I was talking to John, who's the boss of a marketing company, and he was explaining to me about a machine learning model they tried to use to find out which customers might stop using their service. </p><p>The idea was to offer those customers special deals and things to make them stay. But the model didn't work as well as planned. It mistakenly labelled some customers as likely to leave when they weren't really planning to. </p><p>So, those customers got special treatment for a while until the model recalibrated them as not being risky! The company stopped offering the free stuff now. A major portion of such customers left as they felt ignored.</p><p>The same thing happened with the police department in the city. They started using a machine learning model to figure out where to put more police officers and support staff in areas where crimes were happening. </p><p>But the model wasn't perfect either. It identified some areas as being high risk when they really weren't and sent too many patrols there. Meanwhile, other areas that were risky didn't get enough attention, and the crime rate increased unexpectedly!</p><h3><strong>Sub-optimal business processes</strong></h3><p>Organizations aim to deploy an IT-enabled intelligent end-to-end business process &#8211; giving the best customer experience. However, the most ignored part is the current business process. Is it optimal, customer-centric, and stable under various uncertain settings?</p><div class="pullquote"><p>Typically,<em> automation increases the efficiency of an efficient business process and the inefficiency of an inefficient business process</em></p></div><p>My friend Patrik leads an advertising company. The company recently started using web analytics and a high-end optimization engine to show the right product(s) to the right customer(s). However, the problem was that the way they were collecting data didn't match up with how customers actually behaved in the portal. Moreover, they didn&#8217;t have an A/B testing process.</p><p>With automation, the ML models built on misaligned data started damaging customer experience everywhere and ended up with a massive failure!</p><p>Thus, a business with sub-optimal or poor business processes, when automated with IT and sophisticated data technologies, can only expect serious bottlenecks during IT-enabled execution. Hence, the original idea to deploy cutting-edge technology and IT automation to scale up the business would fall flat on the ground. The cobra-effect!</p><h3><strong>Duplicated investment</strong></h3><p>Many organizations invest heavily in enterprise technologies, sophisticated hardware, and software. For example, a large retail company might invest in a customer data platform, a machine learning tool for demand forecasting, and an inventory management system. </p><p>However, in a large enterprise, there are often different data science teams working on similar areas in silos. In the absence of proper coordination, the teams can end up duplicating technology or tool investments.</p><p>To make it worse, in many cases, the technologies are not even used properly due to a lack of awareness and willingness to change. From an overall organizational point of view, this significantly reduces the ROI. And this creates a barrier to the adoption of future technologies, continuing the cycle of ineffective investments.</p><h3><strong>Legacy vs emerging systems</strong></h3><p>New technologies like AI/ML, IoT, Blockchain, and Cloud are the buzzwords in the business world. Many companies are investing big bucks into them, even acquiring start-ups that specialize in these areas. But, unfortunately, the return on investment (ROI) can be quite low or even negative in some cases due to conflicts with existing legacy systems.</p><p>For example, let's say a company invested in IoT systems to gain an edge in data-driven insights. However, their 25-year-old systems can't handle the massive amount of streaming data from IoT devices. This conflict creates problems with the traditional skill set of the company and a lack of proper governance. If the conflict is not resolved effectively, the investment could end up causing more harm than good.</p><p>It's just like the cobra effect in action once again!</p><h3><strong>The real Cobra Effect</strong></h3><p>Now, let's talk about ethics, morality, societal impact, and sustainability.</p><p>Data Science and the latest AI/ML systems are programmed to make decisions based solely on data and algorithms. They don't have a sense of right and wrong, and can only make decisions based on what they are programmed to do.</p><p>Hence, it is easy to create biased, incorrect, and myopic viewpoints. For example, it might prioritize profits over human lives, the environment, and other moral considerations. Ethics and morality can be compromised big time with fake and unsolicited information. This is a slippery path that could lead to a world where moral values are no longer prioritized or respected.</p><p>Next, social systems can also be negatively affected by AI/ML. As automated AI/ML continues to increase, it's likely that many jobs may become obsolete, leading to unemployment and potentially widening the gap between the rich and poor. Additionally, social media algorithms can lead to polarization and division among different groups of people.</p><p>Finally, the environment is another area where AI/ML has severe negative impacts. The energy required to train and operate AI/ML models can significantly increase carbon footprints and greenhouse gas emissions and other environmental problems.</p><p>For example, the latest LLMs are the largest Neural Network models with trillions of parameters. And training such models consumes a lot of energy. For example, the carbon footprint for training GPT-3 is <a href="https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/">estimated</a> to be the same as driving to the moon and coming back! </p><p>In short, our short-term gains are compromised big time with overall sustainability, leading to serious damage to our planet.</p><h3><strong>Conclusion</strong></h3><p>To summarize, data-driven emerging technologies are there to help enterprises to fly. However, a good intention to adopt such technologies is just not enough. Organizations need to do a few checks internally before they sign up with such ideas:</p><ul><li><p>Seamless data integration with a single source of truth</p></li><li><p>Define and optimize business processes keeping the focus on the end customers</p></li><li><p>Manage conflicts among internal business divisions. The best way to tackle it is to look for global objectives rather than myopic views on short-term local gains</p></li><li><p>Manage conflicts between emerging and legacy systems. The best way to resolve this will be to foster an innovation culture with wide acceptability</p></li><li><p>Be logical and aim to achieve big but only with small and frequent steps</p></li><li><p>And, most importantly, the environment and sustainability</p></li></ul><p>The above steps when done correctly must negate the cobra effects. It cannot happen in a single day. </p><blockquote><p><strong>Only a good intention with the right focus, a well-thought strategy, and collaboration across all levels will win in the end.</strong></p></blockquote><p>Have you seen any other cobra effects? </p><p>Please let me know.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.datascienzz.com/p/the-cobra-effect-in-data-science/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.datascienzz.com/p/the-cobra-effect-in-data-science/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item></channel></rss>