Jekyll2020-05-03T04:09:55+00:00https://quanpr.github.io/feed.xmlPengrui QuanAn ongoing master student at ECE department, UC Los Angeles. Obtained his B.Sc Degree, First Class, from the Chinese Univeristy of Hong Kong.
Pengrui QuanPractical implementations of Newton’s method on Deep Neural Network (in progress)2019-12-01T00:00:00+00:002019-12-01T00:00:00+00:00https://quanpr.github.io/2019/12/01/Second-order-optimization-on-DNNs<p>Stochastic gradieng (SG) methods are currently the major optimization method being used for deep learning because of the simplicity and efficiency. However, the method is sometimes sensitive to parameters and their tuning can be a painful process. To alleviate the above issues, we study the Newton’s method for training Deep Neural Networks (DNN), which can not only diminish the need for hyper-parameter tuning but also emjoy the benefit of fast convergence for convex functions.</p>
<p>We have developed a <a href="https://github.com/cjlin1/simpleNN">toolkit</a> in Tensorflow and MATLAB for DNNs using Newton-CG methods. For the core operation of Gauss-Newton matrix-vector products, I use Tensorflow’s vector-Jacobian products. See implementation details in this <a href="/assets/pdf/Calculating_Gauss_Newton_Matrix_Vector_product_by_Vector_Jacobian_Products.pdf">document</a>.</p>
<p><img src="/assets/images/accu_3layers.png" alt="Accuracy on 3-layer CNN" /></p>
<p style="text-align: center;"><em>Accuracy on 3-layer CNN</em></p>
<p><img src="/assets/images/accu_7layers.png" alt="Accuracy on 7-layer CNN" /></p>
<p style="text-align: center;"><em>Accuracy on 7-layer CNN</em></p>
<p>I also gave a <a href="/assets/pdf/Newton_methods.pdf">presentation</a> on this project. Codes are available at <a href="https://github.com/cjlin1/simpleNN">SimpleNN</a>.</p>Pengrui QuanStochastic gradieng (SG) methods are currently the major optimization method being used for deep learning because of the simplicity and efficiency. However, the method is sometimes sensitive to parameters and their tuning can be a painful process. To alleviate the above issues, we study the Newton’s method for training Deep Neural Networks (DNN), which can not only diminish the need for hyper-parameter tuning but also emjoy the benefit of fast convergence for convex functions.Black-box hard-label GenAttack2019-09-27T00:00:00+00:002019-09-27T00:00:00+00:00https://quanpr.github.io/2019/09/27/Hard-label-GenAttack<p>Existing GenAttack is a state-of-the-art methods on fooling deep learning based image classifier in black-box setting. However, it still requires probability after softmax layer for adversarial samples selection. In this work, we extend the GenAttack to hard-label case. With an appropriate amount of queries to the TOP-1 class label, the GenAttack can attack the Inception based image classifier on ImageNet.</p>
<p style="text-align: center;"><img src="/assets/images/hard-label.png" alt="L2 distance changes w.r.t. the number of queries" />
<em>L2 distance changes w.r.t. the number of queries</em></p>
<p style="text-align: center;"><img src="/assets/images/compared.png" alt="Fooling the image classifier by classifying squirrel as parking meter" />
<em>Fooling the image classifier by classifying squirrel as parking meter</em></p>Pengrui QuanExisting GenAttack is a state-of-the-art methods on fooling deep learning based image classifier in black-box setting. However, it still requires probability after softmax layer for adversarial samples selection. In this work, we extend the GenAttack to hard-label case. With an appropriate amount of queries to the TOP-1 class label, the GenAttack can attack the Inception based image classifier on ImageNet.Reviews on second order optimization methods for deep learning2019-05-16T00:00:00+00:002019-05-16T00:00:00+00:00https://quanpr.github.io/2019/05/16/Reviews%20on%20second-order-optimization-on-DNN<p>I mainly inverstigate different second-order optimization methods in DNN in a stochastic manner, including Trust-region Method (TR) and (adaptive) Cubic Reg- ularization (CR), and compared their convergence rate, generalization error and computation efficiency with SGD.</p>
<p style="text-align: center;"><img src="/assets/images/loss-srelu-resnet.png" alt="Training loss of SReLU-ResNet" /><em>Training loss of SReLU-ResNet</em></p>
<p style="text-align: center;"><img src="/assets/images/accu-srelu-resnet.png" alt="Test accuracy of SReLU-ResNet" /><em>Test accuracy of SReLU-ResNet</em></p>
<p>I use the gradient based solver to solve each the inner optimization problems for both TR and CR problem. The method can effectively converge to a local minimum that has similar performance as SGD. However the overhead in each step is significant. I also compared LBFGS and Newton-CG methods in the project. The inferior performance may be caused by the noisy estimation of Hessian term.</p>
<p>Codes are available at <a href="https://github.com/quanpr/Second-order-method-for-Deep-Neural-Network.">Github</a>. Experiment details are summarized in the <a href="/assets/pdf/ECE_236C_course_project_report_805227042_Pengrui.pdf">report</a>.</p>Pengrui QuanI mainly inverstigate different second-order optimization methods in DNN in a stochastic manner, including Trust-region Method (TR) and (adaptive) Cubic Reg- ularization (CR), and compared their convergence rate, generalization error and computation efficiency with SGD.Decouple ROI pooling for object detection2019-03-20T00:00:00+00:002019-03-20T00:00:00+00:00https://quanpr.github.io/2019/03/20/Decouple-ROI-pooling-for-object-detection<p>I mainly focus on region-based object detection frameworks (i.e. Faster-RCNN) which are composed of proposal modules on top of feature extraction modules and make detections. Based on the work of current region-based CNN detection, I introduced an approach to enhance the adaptive receptive capability of CNN, namely decouple ROI pooling. The module can enable the network to select the best receptive field to improve its classification accuracy without sacrificing its ability of bounding box regression.</p>
<p style="text-align: center;"><img src="/assets/images/module.png" alt="Decouple ROI modules" /><em>Decouple ROI modules</em></p>
<p>Codes are available at <a href="https://github.com/quanpr/decouple-roi-pooling">Github</a>. Experiment details are summarized in the <a href="/assets/pdf/CS_260_group_project_report.pdf">report</a>.</p>Pengrui QuanI mainly focus on region-based object detection frameworks (i.e. Faster-RCNN) which are composed of proposal modules on top of feature extraction modules and make detections. Based on the work of current region-based CNN detection, I introduced an approach to enhance the adaptive receptive capability of CNN, namely decouple ROI pooling. The module can enable the network to select the best receptive field to improve its classification accuracy without sacrificing its ability of bounding box regression.Deep-learning based Vehicle Re-identification2018-04-11T00:00:00+00:002018-04-11T00:00:00+00:00https://quanpr.github.io/2018/04/11/Deep-Learning-based-Vehicle-Reidentification<p>Vehicle re-identification (ReID) is an important problem and has many applications in video surveillance and intelligent transportation. The existing approaches to vehicle ReID mainly relies on the appearance information of vehicles with an oversimplified spatiotemporal information. In this project, we make use of both the appearance of vehicles and their time and geo-location relationship to improve the retrieved accuracy.</p>
<table>
<thead>
<tr>
<th>Method</th>
<th>mAP (%)</th>
<th>top 1 accuracy (%)</th>
<th>top 5 accuracy (%)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Single ResNet-18</td>
<td>33.5</td>
<td>77.0</td>
<td>83.3</td>
</tr>
<tr>
<td>Single ResNet-50</td>
<td>37.1</td>
<td>79.5</td>
<td>87.4</td>
</tr>
<tr>
<td>SNN (ResNet-50)</td>
<td>41.1</td>
<td>89.4</td>
<td>95.2</td>
</tr>
<tr>
<td>SNN (ResNet-18) + Spatiotemporal model</td>
<td>42.4</td>
<td>89.9</td>
<td>96.3</td>
</tr>
<tr>
<td>SNN (ResNet-50) + Spatiotemporal model</td>
<td>47.6</td>
<td>94.7</td>
<td>99.5</td>
</tr>
<tr>
<td>Integrated SNN (ResNet-18)</td>
<td>49.7</td>
<td>94.8</td>
<td>100</td>
</tr>
<tr>
<td>Integrated SNN (ResNet-50)</td>
<td><strong>58.6</strong></td>
<td><strong>96.6</strong></td>
<td><strong>100</strong></td>
</tr>
</tbody>
</table>
<p>Implementations and experiment details may be found in the <a href="/assets/pdf/1155092203_QuanPengrui.pdf">report</a>.</p>Pengrui QuanVehicle re-identification (ReID) is an important problem and has many applications in video surveillance and intelligent transportation. The existing approaches to vehicle ReID mainly relies on the appearance information of vehicles with an oversimplified spatiotemporal information. In this project, we make use of both the appearance of vehicles and their time and geo-location relationship to improve the retrieved accuracy.My Example Post2016-05-20T00:00:00+00:002016-05-20T00:00:00+00:00https://quanpr.github.io/2016/05/20/my-example-post<p>Eos eu docendi tractatos sapientem, brute option menandri in vix, quando vivendo accommodare te ius. Nec melius fastidii constituam id, viderer theophrastus ad sit, hinc semper periculis cum id. Noluisse postulant assentior est in, no choro sadipscing repudiandae vix. Vis in euismod delenit dignissim. Ex quod nostrum sit, suas decore animal id ius, nobis solet detracto quo te.</p>
<p>No laudem altera adolescens has, volumus lucilius eum no. Eam ei nulla audiam efficiantur. Suas affert per no, ei tale nibh sea. Sea ne magna harum, in denique scriptorem sea, cetero alienum tibique ei eos. Labores persequeris referrentur eos ei.</p>Pengrui QuanEos eu docendi tractatos sapientem, brute option menandri in vix, quando vivendo accommodare te ius. Nec melius fastidii constituam id, viderer theophrastus ad sit, hinc semper periculis cum id. Noluisse postulant assentior est in, no choro sadipscing repudiandae vix. Vis in euismod delenit dignissim. Ex quod nostrum sit, suas decore animal id ius, nobis solet detracto quo te.Some articles are just so long they deserve a really long title to see if things will break well2016-05-20T00:00:00+00:002016-05-20T00:00:00+00:00https://quanpr.github.io/misc/2016/05/20/super-long-article<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit. Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum. In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris. Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc. Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit. Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum. In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris. Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc. Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit. Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum. In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris. Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc. Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>Pengrui QuanLorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit. Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum. In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris. Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc. Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.This post demonstrates post content styles2016-05-20T00:00:00+00:002016-05-20T00:00:00+00:00https://quanpr.github.io/junk/2016/05/20/this-post-demonstrates-post-content-styles<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit.</p>
<h2 id="some-great-heading-h2">Some great heading (h2)</h2>
<p>Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum. In arcu magna, aliquet vel pretium et, molestie et arcu.</p>
<p>Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris. Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc. Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>
<h2 id="another-great-heading-h2">Another great heading (h2)</h2>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit.</p>
<h3 id="some-great-subheading-h3">Some great subheading (h3)</h3>
<p>Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum. In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum.</p>
<p>Phasellus et hendrerit mauris. Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc.</p>
<h3 id="some-great-subheading-h3-1">Some great subheading (h3)</h3>
<p>Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>
<blockquote>
<p>This quote will change your life. It will reveal the secrets of the universe, and all the wonders of humanity. Don’t misuse it.</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt.</p>
<h3 id="some-great-subheading-h3-2">Some great subheading (h3)</h3>
<p>Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit. Proin convallis mi ac felis pharetra aliquam. Curabitur dignissim accumsan rutrum.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><p></span>Hello, World!<span class="nt"></p></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris.</p>
<h4 id="you-might-want-a-sub-subheading-h4">You might want a sub-subheading (h4)</h4>
<p>In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris.</p>
<p>In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris.</p>
<h4 id="but-its-probably-overkill-h4">But it’s probably overkill (h4)</h4>
<p>In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris.</p>
<h3 id="oh-hai-an-unordered-list">Oh hai, an unordered list!!</h3>
<p>In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris.</p>
<ul>
<li>First item, yo</li>
<li>Second item, dawg</li>
<li>Third item, what what?!</li>
<li>Fourth item, fo sheezy my neezy</li>
</ul>
<h3 id="oh-hai-an-ordered-list">Oh hai, an ordered list!!</h3>
<p>In arcu magna, aliquet vel pretium et, molestie et arcu. Mauris lobortis nulla et felis ullamcorper bibendum. Phasellus et hendrerit mauris.</p>
<ol>
<li>First item, yo</li>
<li>Second item, dawg</li>
<li>Third item, what what?!</li>
<li>Fourth item, fo sheezy my neezy</li>
</ol>
<h2 id="headings-are-cool-h2">Headings are cool! (h2)</h2>
<p>Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc. Praesent varius interdum vehicula. Aenean risus libero, placerat at vestibulum eget, ultricies eu enim. Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>
<p>Praesent nulla tortor, malesuada adipiscing adipiscing sollicitudin, adipiscing eget est.</p>
<p>Proin eget nibh a massa vestibulum pretium. Suspendisse eu nisl a ante aliquet bibendum quis a nunc.</p>
<h3 id="tables">Tables</h3>
<table>
<thead>
<tr>
<th>Title 1</th>
<th>Title 2</th>
<th>Title 3</th>
<th>Title 4</th>
</tr>
</thead>
<tbody>
<tr>
<td>lorem</td>
<td>lorem ipsum</td>
<td>lorem ipsum dolor</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Title 1</th>
<th>Title 2</th>
<th>Title 3</th>
<th>Title 4</th>
</tr>
</thead>
<tbody>
<tr>
<td>lorem</td>
<td>lorem ipsum</td>
<td>lorem ipsum dolor</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit amet</td>
<td>lorem ipsum dolor sit amet consectetur</td>
<td>lorem ipsum dolor sit amet</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor</td>
<td>lorem ipsum</td>
<td>lorem</td>
<td>lorem ipsum</td>
</tr>
<tr>
<td>lorem ipsum dolor</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit amet</td>
<td>lorem ipsum dolor sit amet consectetur</td>
</tr>
</tbody>
</table>Bart SimpsonLorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce bibendum neque eget nunc mattis eu sollicitudin enim tincidunt. Vestibulum lacus tortor, ultricies id dignissim ac, bibendum in velit.Welcome To Jekyll2016-05-20T00:00:00+00:002016-05-20T00:00:00+00:00https://quanpr.github.io/2016/05/20/welcome-to-jekyll<p>You’ll find this post in your <code class="language-plaintext highlighter-rouge">_posts</code> directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run <code class="language-plaintext highlighter-rouge">jekyll serve</code>, which launches a web server and auto-regenerates your site when a file is updated.</p>
<p>To add new posts, simply add a file in the <code class="language-plaintext highlighter-rouge">_posts</code> directory that follows the convention <code class="language-plaintext highlighter-rouge">YYYY-MM-DD-name-of-post.ext</code> and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.</p>
<p>Jekyll also offers powerful support for code snippets:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">print_hi</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"Hi, </span><span class="si">#{</span><span class="nb">name</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="n">print_hi</span><span class="p">(</span><span class="s1">'Tom'</span><span class="p">)</span>
<span class="c1">#=> prints 'Hi, Tom' to STDOUT.</span></code></pre></figure>
<p>Check out the <a href="http://jekyllrb.com/docs/home">Jekyll docs</a> for more info on how to get the most out of Jekyll. File all bugs/feature requests at <a href="https://github.com/jekyll/jekyll">Jekyll’s GitHub repo</a>. If you have questions, you can ask them on <a href="https://talk.jekyllrb.com/">Jekyll Talk</a>.</p>Pengrui QuanYou’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run jekyll serve, which launches a web server and auto-regenerates your site when a file is updated.Some articles are just so short that we have to make the footer stick2016-05-19T00:00:00+00:002016-05-19T00:00:00+00:00https://quanpr.github.io/misc/2016/05/19/super-short-article<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>Pengrui QuanLorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.