1
0
DavidZhang73.github.io/SVD-for-homogeneous-linear-equation/index.html

490 lines
21 KiB
HTML
Raw Normal View History

2020-06-05 00:15:41 +08:00
<!DOCTYPE html>
<html lang="zh-CN" class="loading">
<head>
<!-- hexo-inject:begin --><!-- hexo-inject:end --><meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
<meta name="viewport" content="width=device-width, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no">
<title>想法速记SVD求齐次线性方程组 Ax=0 的解 - Blog</title>
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
<meta name="google" content="notranslate" />
<meta name="keywords" content="Developer, Python, C++, JavaScript, Java,">
<meta name="description" content="DavidZ&#39;s Blog,
原创发表于 DavidZ Blog遵循 CC 4.0 BY-NC-SA 版权协议,转载请附上原文出处链接及本声明。
众所周知SVD(奇异值分解)可以用于最小二乘法求齐次线性方程组$A\vec,">
<meta name="author" content="DavidZ">
<link rel="alternative" href="atom.xml" title="Blog" type="application/atom+xml">
<link rel="icon" href="/img/favicon.png">
<link href="https://fonts.loli.net/css?family=Roboto+Mono|Rubik&display=swap" rel="stylesheet">
<link rel="stylesheet" href="//at.alicdn.com/t/font_1429596_nzgqgvnmkjb.css">
<link rel="stylesheet" href="//cdn.bootcss.com/animate.css/3.7.2/animate.min.css">
<link rel="stylesheet" href="//cdn.bootcss.com/social-share.js/1.0.16/css/share.min.css">
<link rel="stylesheet" href="//cdn.bootcss.com/codemirror/5.48.4/codemirror.min.css">
<link rel="stylesheet" href="//cdn.bootcss.com/codemirror/5.48.4/theme/dracula.css">
<link rel="stylesheet" href="/css/obsidian.css">
<link rel="stylesheet" href="/css/ball-atom.min.css">
<meta name="generator" content="Hexo 4.2.0"><!-- hexo-inject:begin --><!-- hexo-inject:end --></head>
<body class="loading">
<!-- hexo-inject:begin --><!-- hexo-inject:end --><div class="loader">
<div class="la-ball-atom la-2x">
<div></div>
<div></div>
<div></div>
<div></div>
</div>
</div>
<span id="config-title" style="display:none">Blog</span>
<div id="loader"></div>
<div id="single">
<div class="scrollbar gradient-bg-rev"></div>
<div id="top" style="display: block;">
<div class="bar" style="width: 0;"></div>
<div class="navigation animated fadeIn fast delay-1s">
<img id="home-icon" class="icon-home" src="/img/favicon.png" alt="" data-url="https://blog.davidz.cn">
<div id="play-icon" title="Play/Pause" class="iconfont icon-play"></div>
<h3 class="subtitle">想法速记SVD求齐次线性方程组 Ax=0 的解</h3>
<div class="social">
<!-- <div class="like-icon">-->
<!-- <a href="javascript:;" class="likeThis active"><span class="icon-like"></span><span class="count">76</span></a>-->
<!-- </div>-->
<div>
<div class="share">
<a href="javascript:;" class="iconfont icon-share1"></a>
<div class="share-component-cc" data-disabled="facebook,douban,linkedin,diandian,tencent,google"></div>
</div>
</div>
</div>
</div>
</div>
<div class="section">
<div class=article-header-wrapper>
<div class="article-header">
<div class="article-cover animated fadeIn" style="
animation-delay: 600ms;
animation-duration: 1.2s;
background-image:
radial-gradient(ellipse closest-side, rgba(0, 0, 0, 0.65), #100e17),
2020-10-09 10:48:26 +08:00
url(https://davidz-blog.oss-cn-beijing.aliyuncs.com/img/idea-1602143497.jpg) ">
2020-06-05 00:15:41 +08:00
</div>
<div class="else">
<p class="animated fadeInDown">
<a href="/categories/Idea"><b>
</b>IDEA<b></b></a>
六月 04, 2020
</p>
<h3 class="post-title animated fadeInDown"><a href="/SVD-for-homogeneous-linear-equation/" title="想法速记SVD求齐次线性方程组 Ax&#x3D;0 的解" class="">想法速记SVD求齐次线性方程组 Ax&#x3D;0 的解</a>
</h3>
<p class="post-count animated fadeInDown">
<span>
<b class="iconfont icon-text2"></b> <i>文章字数</i>
2.9k
</span>
<span>
<b class="iconfont icon-timer__s"></b> <i>阅读约需</i>
3 mins.
</span>
<span id="busuanzi_container_page_pv">
<b class="iconfont icon-read"></b> <i>阅读次数</i>
<span id="busuanzi_value_page_pv">0</span>
</span>
</p>
</div>
</div>
</div>
<div class="screen-gradient-after">
<div class="screen-gradient-content">
<div class="screen-gradient-content-inside">
<div class="bold-underline-links screen-gradient-sponsor">
<p>
<span class="animated fadeIn delay-1s"></span>
</p>
</div>
</div>
</div>
</div>
<div class="article">
<div class='main'>
<div class="content markdown animated fadeIn">
<blockquote>
<p>原创发表于 <a href="https://blog.davidz.cn">DavidZ Blog</a>,遵循 <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode" target="_blank" rel="noopener">CC 4.0 BY-NC-SA</a> 版权协议,转载请附上原文出处链接及本声明。</p>
</blockquote>
<p>众所周知SVD(奇异值分解)可以用于最小二乘法求齐次线性方程组$A\vec{x}=\vec{0}$的解。我看了很多资料,大多使用数学公式推导,得出结论。但是,曾经线性代数差点挂科的我,总觉得有些蹊跷。想了两天,终于有了一些感性的认知(不一定是对的😂),赶紧记录下来。</p>
<h2 id="矩阵有何意义"><a class="header-anchor" href="#矩阵有何意义"></a>矩阵有何意义</h2>
<p>按照我的理解,一个矩阵的实际意义是对应一个线性变换,这个变换可以理解为瞬间运动。例如,一个旋转矩阵,</p>
<p>$$<br>
A=\begin{bmatrix}<br>
cos\theta &amp; sin\theta \\\<br>
-sin\theta &amp; cos\theta<br>
\end{bmatrix}<br>
$$</p>
<p>它的意思是,把一个向量顺时针旋转$\theta$。也就是说,给定一个$\vec{v_1}=[-1, 1]^T$, 那么变换的结果就是$\vec{v_2}=A\vec{v_1}=[1, 1]^T$.</p>
<p>除了旋转,矩阵还可以表示包括缩放,投影在内的所有<strong>线性变换</strong></p>
<p>十分推荐大家去看 3Blue1Brown 的 <em>线性代数的本质</em>B 站有<a href="https://www.bilibili.com/video/BV1ys411472E" target="_blank" rel="noopener">官方翻译版</a>,它完全颠覆了我对线性代数的认知。</p>
<h2 id="SVD到底干了什么"><a class="header-anchor" href="#SVD到底干了什么"></a>SVD到底干了什么</h2>
<p>$$<br>
A = U\Sigma V^T<br>
$$</p>
<p>SVD把矩阵$A(m\times n)$分解成了,</p>
<ul>
<li>$U(m\times m)$: 左奇异矩阵</li>
<li>$\Sigma(m\times n)$: 奇异值矩阵</li>
<li>$V(n\times n)$: 右奇异矩阵</li>
</ul>
<p>重点来了SVD的意思就是把一个本来由矩阵$A$表示的变换,转化成一个由$U,\Sigma,V$表示的变换。这个变换是,把一个向量,从以$V$为基向量的空间线性变换到成以$U$为基向量的空间中去($\Sigma$的意义可以说是缩放,待证实,暂时忽略)。这样我们就可以更深入的理解这个变换了。</p>
<p>例如旋转$\vec{v_1}$90度得到$\vec{v_2}$</p>
<p><img src="//davidz.cn/static/blog/2020-06-04-SVD-for-homogeneous-linear-equation/rotate90.svg" alt="rotate 90"></p>
<p>其中,</p>
<p>$$<br>
U=\begin{bmatrix}<br>
0 &amp; 1 \\\<br>
1 &amp; 0<br>
\end{bmatrix}<br>
$$</p>
<p>$$<br>
\Sigma=\begin{bmatrix}<br>
1 &amp; 1<br>
\end{bmatrix}<br>
$$</p>
<p>$$<br>
V=\begin{bmatrix}<br>
-1 &amp; 0 \\\<br>
0 &amp; 1<br>
\end{bmatrix}<br>
$$</p>
<p>也就是说,矩阵$A$可以被理解为,我们把一个向量$\vec{v_1}$,从以$\vec{e_1}=[-1, 0]^T,\vec{e_2}=[0, 1]^T$为基向量的空间线性变换到了以$\vec{e_1}=[0, 1]^T,\vec{e_2}=[1, 0]^T$为基向量的空间中。这个变换表现为旋转了90度。</p>
<h2 id="所以如何理解"><a class="header-anchor" href="#所以如何理解"></a>所以如何理解</h2>
<p>说回求齐次线性方程组$Ax=0$的解来。</p>
<p>按照矩阵的意义,我们这里要求的是,已知一个线性变换$A$,给定一个$\vec{x}$,使得线性变换后的结果为$\vec{0}$。</p>
<p>此时非常重要的是,如果$x=\vec{0}$,那一定成立,但是我们想找一个非平凡的解。</p>
<p>我们暂时不关心这个解是否存在也就是说如果不存在我们就找个最接近的最小二乘法思想我们直接使用SVD分解矩阵$A$,得到对应的$U,\Sigma,V$。</p>
<p>按照SVD的作用我们现在可以说矩阵$A$这个线性变换,把一个$\vec{x}$,从以$V$为基向量的空间线性变换到了以$U$为基向量的空间中,而我们想找,在以$V$为基向量的空间中,哪个向量会在投影后趋近于或者等于$\vec{0}$。更重要的是,我们只在乎这个向量的方向,而不在乎他的大小,因为它等于$\vec{0}$是个平凡解,这就像最小二乘法中,我们规定$|\vec{x}|=1$。</p>
<p>这时,答案就开始变得清晰了,因为我们想找的$\vec{x}$,应该就是$V$这组基向量中特异值$\sigma$最小的那一个$\vec{e_{min}}$,也就是说$\vec{x}=\vec{e_{min}}$。此时有两种情况,</p>
<ol>
<li>$\sigma=0$ 那么$\vec{x}$投影后的就是$\vec{0}$。</li>
<li>$\sigma\neq0$,那么$\vec{x}$投影后是使$A\vec{x}$最小的解。因为如果$\vec{x}\neq\vec{e_{min}}$,也就是说它偏离了$\vec{e_{min}}$,那么它一定由$\vec{e_{min}}$和另一个基向量线性组合,而无论怎么组合,$\sigma_{combine}\geq\sigma_{min}$。</li>
</ol>
<p>因此,我们求解$A\vec{x}=0$的过程就是,</p>
<ol>
<li>$U,\Sigma,V^T=SVD(A)$</li>
<li>$\vec{x}=V[:, -1]$</li>
</ol>
<!--[if lt IE 9]><script>document.createElement('audio');</script><![endif]-->
<audio id="audio" loop="1" preload="auto" controls="controls"
data-autoplay="false">
2020-09-07 21:52:27 +08:00
<source type="audio/mpeg" src="https://davidz-blog.oss-cn-beijing.aliyuncs.com/music/Delacey - Dream It Possible.mp3">
2020-06-05 00:15:41 +08:00
</audio>
<div id='gitalk-container' class="comment link"
data-ae='true'
data-ci='489076c5dd3f5ba13f67'
data-cs='d6e3b245787b0b74d0dbe2639ef87f452a401194'
data-r='blog.davidz.cn'
data-o='DavidZhang73'
data-a='DavidZhang73'
data-d=''
>留言</div>
</div>
<div class="sidebar">
<div class="box animated fadeInRight">
<div class="subbox">
2020-09-07 21:52:27 +08:00
<img src="https://davidz-blog.oss-cn-beijing.aliyuncs.com/img/2019-1599483796.jpg" height=300 width=300></img>
2020-06-05 00:15:41 +08:00
<p>DavidZ</p>
<span>凡事都要留几分</span>
<dl>
<dd><a href="https://github.com/DavidZhang73" target="_blank"><span
class=" iconfont icon-github"></span></a></dd>
<dd><a href="https://twitter.com/david731998" target="_blank"><span
class=" iconfont icon-twitter"></span></a></dd>
<dd><a href="https://stackoverflow.com/users/12693553/davidz" target="_blank"><span
class=" iconfont icon-stack-overflow"></span></a></dd>
</dl>
</div>
<ul>
2020-10-09 14:46:16 +08:00
<li><a href="/">15 <p>文章</p></a></li>
2020-10-09 10:48:26 +08:00
<li><a href="/categories">8 <p>分类</p></a></li>
2020-10-09 14:46:16 +08:00
<li><a href="/tags">15 <p>标签</p></a></li>
2020-06-05 00:15:41 +08:00
</ul>
</div>
<div class="box sticky animated fadeInRight faster">
<div id="toc" class="subbox">
<h4>目录</h4>
<ol class="toc"><li class="toc-item toc-level-2"><a class="toc-link" href="#矩阵有何意义"><span class="toc-number">1.</span> <span class="toc-text">¶矩阵有何意义</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#SVD到底干了什么"><span class="toc-number">2.</span> <span class="toc-text">¶SVD到底干了什么</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#所以如何理解"><span class="toc-number">3.</span> <span class="toc-text">¶所以如何理解</span></a></li></ol>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div id="back-to-top" class="animated fadeIn faster">
<div class="flow"></div>
<span class="percentage animated fadeIn faster">0%</span>
<span class="iconfont icon-top02 animated fadeIn faster"></span>
</div><!-- hexo-inject:begin --><!-- hexo-inject:end -->
</body>
<footer>
<p class="copyright" id="copyright">
&copy; 2020
<span class="gradient-text">
DavidZ
</span>.
Powered by <a href="http://hexo.io/" title="Hexo" target="_blank" rel="noopener">Hexo</a>
Theme
<span class="gradient-text">
<a href="https://github.com/TriDiamond/hexo-theme-obsidian" title="Obsidian" target="_blank" rel="noopener">Obsidian</a>
</span>
2020-07-17 11:27:08 +08:00
<small><a href="https://github.com/TriDiamond/hexo-theme-obsidian/blob/master/CHANGELOG.md" title="v1.4.4" target="_blank" rel="noopener">v1.4.4</a></small>
2020-06-05 00:15:41 +08:00
</p>
</footer>
2020-06-05 12:12:10 +08:00
<script type="text/javascript" src="https://cdn.bootcss.com/mathjax/2.7.7/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
2020-06-05 00:15:41 +08:00
</script>
<script>
MathJax.Hub.Config({
"HTML-CSS": {
preferredFont: "TeX",
availableFonts: ["STIX", "TeX"],
linebreaks: {
automatic: true
},
EqnChunk: (MathJax.Hub.Browser.isMobile ? 10 : 50)
},
tex2jax: {
inlineMath: [
["$", "$"],
["\\(", "\\)"]
],
processEscapes: true,
ignoreClass: "tex2jax_ignore|dno",
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code']
},
TeX: {
noUndefined: {
attributes: {
mathcolor: "red",
mathbackground: "#FFEEEE",
mathsize: "90%"
}
},
Macros: {
href: "{}"
}
},
messageStyle: "none"
});
</script>
<script>
function initialMathJax() {
MathJax.Hub.Queue(function () {
var all = MathJax.Hub.getAllJax(),
i;
// console.log(all);
for (i = 0; i < all.length; i += 1) {
all[i].SourceElement().parentNode.className += ' has-jax';
}
});
}
function reprocessMathJax() {
if (typeof MathJax !== 'undefined') {
MathJax.Hub.Queue(["Typeset", MathJax.Hub]);
}
}
</script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<link rel="stylesheet" href="//cdn.bootcss.com/gitalk/1.6.2/gitalk.min.css">
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/gitalk/1.6.2/gitalk.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 00:15:41 +08:00
<script src="//cdn.bootcss.com/jquery/3.4.1/jquery.min.js"></script>
<script src="/js/plugin.js"></script>
<script src="/js/obsidian.js"></script>
<script src="/js/jquery.truncate.js"></script>
<script src="/js/search.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 00:15:41 +08:00
<script src="//cdn.bootcss.com/typed.js/2.0.10/typed.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 00:15:41 +08:00
<script src="//cdn.bootcss.com/blueimp-md5/2.12.0/js/md5.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 00:15:41 +08:00
<script src="//cdn.bootcss.com/social-share.js/1.0.16/js/social-share.min.js"></script>
2020-06-05 12:12:10 +08:00
<script src="https://cdn.bootcss.com/codemirror/5.54.0/codemirror.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/javascript/javascript.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/css/css.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/xml/xml.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/htmlmixed/htmlmixed.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/clike/clike.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/php/php.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/shell/shell.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/python/python.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/cmake/cmake.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/powershell/powershell.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 12:12:10 +08:00
<script src="//cdn.bootcss.com/codemirror/5.54.0/mode/yaml/yaml.min.js"></script>
2020-07-17 11:27:08 +08:00
2020-06-05 00:15:41 +08:00
<script src="/js/busuanzi.min.js"></script>
2020-07-17 11:27:08 +08:00
<script>
$(document).ready(function () {
if ($('span[id^="busuanzi_"]').length) {
initialBusuanzi();
}
});
</script>
2020-06-05 00:15:41 +08:00
<link rel="stylesheet" href="//cdn.bootcss.com/photoswipe/4.1.3/photoswipe.min.css">
<link rel="stylesheet" href="//cdn.bootcss.com/photoswipe/4.1.3/default-skin/default-skin.min.css">
<script src="//cdn.bootcss.com/photoswipe/4.1.3/photoswipe.min.js"></script>
<script src="//cdn.bootcss.com/photoswipe/4.1.3/photoswipe-ui-default.min.js"></script>
<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">
<!-- Background of PhotoSwipe.
It's a separate element as animating opacity is faster than rgba(). -->
<div class="pswp__bg"></div>
<!-- Slides wrapper with overflow:hidden. -->
<div class="pswp__scroll-wrap">
<!-- Container that holds slides.
PhotoSwipe keeps only 3 of them in the DOM to save memory.
Don't modify these 3 pswp__item elements, data is added later on. -->
<div class="pswp__container">
<div class="pswp__item"></div>
<div class="pswp__item"></div>
<div class="pswp__item"></div>
</div>
<!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
<div class="pswp__ui pswp__ui--hidden">
<div class="pswp__top-bar">
<!-- Controls are self-explanatory. Order can be changed. -->
<div class="pswp__counter"></div>
<button class="pswp__button pswp__button--close" title="Close (Esc)"></button>
<button class="pswp__button pswp__button--share" title="Share"></button>
<button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>
<button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>
<!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
<!-- element will get class pswp__preloader--active when preloader is running -->
<div class="pswp__preloader">
<div class="pswp__preloader__icn">
<div class="pswp__preloader__cut">
<div class="pswp__preloader__donut"></div>
</div>
</div>
</div>
</div>
<div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
<div class="pswp__share-tooltip"></div>
</div>
<button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
</button>
<button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
</button>
<div class="pswp__caption">
<div class="pswp__caption__center"></div>
</div>
</div>
</div>
</div>
2020-07-17 11:27:08 +08:00
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="//www.googletagmanager.com/gtag/js?id=UA-157733505-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
gtag('js', new Date());
2020-06-05 00:15:41 +08:00
2020-07-17 11:27:08 +08:00
gtag('config', 'UA-157733505-1');
</script>
2020-06-05 00:15:41 +08:00
<script>
2020-07-17 11:27:08 +08:00
function initialTyped() {
var typedTextEl = $('.typed-text');
if (typedTextEl && typedTextEl.length > 0) {
var typed = new Typed('.typed-text', {
strings: ['凡事都要留几分', '颜值是第一生产力'],
typeSpeed: 90,
loop: true,
loopCount: Infinity,
backSpeed: 20,
});
2020-06-05 00:15:41 +08:00
}
2020-07-17 11:27:08 +08:00
}
2020-06-05 00:15:41 +08:00
2020-07-17 11:27:08 +08:00
if ($('.article-header') && $('.article-header').length) {
$(document).ready(function () {
initialTyped();
});
}
2020-06-05 00:15:41 +08:00
</script>
</html>