Souhail HANFI
2023-01-08T12:39:04+00:00
https://hanfi.github.io/
Souhail Hanfi
simple Boto examples with S3
2014-08-26T00:00:00+00:00
https://hanfi.github.io//blog/boto-and-s3
<h2 id="requirements">Requirements</h2>
<p>all you need is python, boto and filechunkio for easy multipart-upload</p>
<h2 id="creating-connection">Creating connection</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">boto</span>
<span class="kn">import</span> <span class="nn">boto.s3.connection</span>
<span class="n">access_key</span> <span class="o">=</span> <span class="s">'put your acces key here'</span>
<span class="n">secret_key</span> <span class="o">=</span> <span class="s">'put your secret key here'</span>
<span class="n">con</span> <span class="o">=</span> <span class="n">boto</span><span class="p">.</span><span class="n">connect_s3</span><span class="p">(</span><span class="n">access_key</span><span class="p">,</span><span class="n">secret_key</span><span class="p">)</span></code></pre></figure>
<p>you can create ~/.boto file and add your credentials on it :</p>
<figure class="highlight"><pre><code class="language-ini" data-lang="ini"><span class="nn">[Credentials]</span>
<span class="py">aws_access_key_id</span> <span class="p">=</span> <span class="s">put_your_acces_key-here</span>
<span class="py">aws_secret_access_key</span> <span class="p">=</span> <span class="s">put_your_secret_key_here</span></code></pre></figure>
<p>so now you can create a connection object just by</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">con</span> <span class="o">=</span> <span class="n">boto</span><span class="p">.</span><span class="n">connect_s3</span><span class="p">()</span></code></pre></figure>
<h2 id="listing-buckets">Listing buckets</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">for</span> <span class="n">bucket</span> <span class="ow">in</span> <span class="n">con</span><span class="p">.</span><span class="n">get_all_buckets</span><span class="p">():</span>
<span class="k">print</span> <span class="n">bucket</span><span class="p">.</span><span class="n">name</span></code></pre></figure>
<h2 id="creating-bucket">Creating bucket</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">bucket</span> <span class="o">=</span> <span class="n">con</span><span class="p">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="s">'new_bucket'</span><span class="p">)</span></code></pre></figure>
<h2 id="listing-all-objects-in-a-bucket">Listing all objects in a bucket</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">bucket</span><span class="p">.</span><span class="nb">list</span><span class="p">()</span>
<span class="k">print</span> <span class="n">key</span><span class="p">.</span><span class="n">name</span></code></pre></figure>
<h2 id="deleting-bucket">Deleting Bucket</h2>
<p>bucket must be empty before deleting (no force method implemented in boto)</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">bucket</span><span class="p">.</span><span class="n">delete</span><span class="p">()</span></code></pre></figure>
<h2 id="creating-an-object-from-string">Creating an object from String</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">key</span> <span class="o">=</span> <span class="n">bucket</span><span class="p">.</span><span class="n">new_key</span><span class="p">(</span><span class="s">"test.txt"</span><span class="p">)</span>
<span class="n">key</span><span class="p">.</span><span class="n">set_content_from_string</span><span class="p">(</span><span class="s">"hello world"</span><span class="p">)</span></code></pre></figure>
<h2 id="creating-an-object-from-file-alternative-key-usage">Creating an object from File (alternative key usage)</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">boto.s3.key</span> <span class="kn">import</span> <span class="n">Key</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">Key</span><span class="p">(</span><span class="n">bucket</span><span class="p">)</span>
<span class="n">key</span><span class="p">.</span><span class="n">key</span> <span class="o">=</span> <span class="s">"file"</span>
<span class="n">key</span><span class="p">.</span><span class="n">set_contents_from_filename</span><span class="p">(</span><span class="s">"path_to_file"</span><span class="p">)</span></code></pre></figure>
<h2 id="retrieving-object-as-string">Retrieving object as String</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">key</span><span class="p">.</span><span class="n">get_contents_to_string</span><span class="p">()</span></code></pre></figure>
<h2 id="retrieving-object-in-file">Retrieving object in File</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">key</span><span class="p">.</span><span class="n">get_contents_to_filename</span><span class="p">()</span></code></pre></figure>
<h2 id="s3-multipart-upload-for-big-files-with-retries">S3 Multipart-Upload for big files (with retries)</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">math</span><span class="p">,</span> <span class="n">os</span>
<span class="kn">from</span> <span class="nn">filechunkio</span> <span class="kn">import</span> <span class="n">FileChunkIO</span>
<span class="n">chunk_size</span> <span class="o">=</span> <span class="mi">5</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span> <span class="c1">#5mo
</span><span class="n">file_name</span> <span class="o">=</span> <span class="s">"my_big_file"</span>
<span class="n">file_size</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">stat</span><span class="p">(</span><span class="n">file_name</span><span class="p">).</span><span class="n">st_size</span>
<span class="n">chunk_number</span> <span class="o">=</span> <span class="nb">int</span> <span class="p">(</span><span class="n">math</span><span class="p">.</span><span class="n">ceil</span><span class="p">(</span><span class="n">file_size</span><span class="o">/</span><span class="n">chunk_size</span><span class="p">))</span><span class="o">+</span><span class="mi">1</span>
<span class="n">mp</span> <span class="o">=</span> <span class="n">bucket</span><span class="p">.</span><span class="n">initiate_multipart_upload</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">basename</span><span class="p">(</span><span class="n">file_name</span><span class="p">))</span>
<span class="n">numberOfRetries</span> <span class="o">=</span> <span class="mi">10</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">chunk_number</span><span class="p">):</span>
<span class="n">offset</span> <span class="o">=</span> <span class="n">chunk_size</span><span class="o">*</span><span class="n">i</span>
<span class="nb">bytes</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">chunk_size</span><span class="p">,</span> <span class="n">file_size</span> <span class="o">-</span> <span class="n">offset</span><span class="p">)</span>
<span class="k">with</span> <span class="n">FileChunkIO</span><span class="p">(</span><span class="n">file_name</span><span class="p">,</span> <span class="s">"r"</span><span class="p">,</span> <span class="n">offset</span><span class="o">=</span><span class="n">offset</span><span class="p">,</span> <span class="nb">bytes</span><span class="o">=</span><span class="nb">bytes</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span> <span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">mp</span><span class="p">.</span><span class="n">upload_part_from_file</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">part_num</span><span class="o">=</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">Exception</span><span class="p">,</span> <span class="n">ex</span><span class="p">:</span>
<span class="k">if</span> <span class="n">numberOfRetries</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ex</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">numberOfRetries</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="n">mp</span><span class="p">.</span><span class="n">upload_part_from_file</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">part_num</span><span class="o">=</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="n">mp</span><span class="p">.</span><span class="n">complete_upload</span><span class="p">()</span></code></pre></figure>
<h2 id="delete-an-object">Delete an object</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">bucket</span><span class="p">.</span><span class="n">delete_key</span><span class="p">(</span><span class="s">'file_to_delete'</span><span class="p">)</span></code></pre></figure>
<h2 id="change-object-acl">Change object ACL</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">key</span><span class="p">.</span><span class="n">set_canned_acl</span><span class="p">(</span><span class="s">'public-read'</span><span class="p">)</span></code></pre></figure>
<h2 id="generate-signed-url-valid-for-30-seconds">Generate signed url valid for 30 seconds</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">signed_url</span> <span class="o">=</span> <span class="n">key</span><span class="p">.</span><span class="n">generate_url</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span></code></pre></figure>
Social integration in Jekyll
2014-08-22T00:00:00+00:00
https://hanfi.github.io//blog/social-integration-in-jekyll
<p>The problem with jekyll, everybody will tell you, it’s the comments.</p>
<p>The biggest advantage of the blogging framework is that it has no need for a Data base, but in the case it becomes it’s weakness.</p>
<p><strong>No Data Base = No Dynamic Data = No Comments</strong></p>
<p>But discuss, facebook and some others provide a commenting sytem that you can integrate to your blog.</p>
<p>Beeing a facebook user, for my blog i choosed the facebook comments integration, and this is how i made it work.</p>
<p>The integration is realy simple :</p>
<p>start by <a href="https://developers.facebook.com/docs/plugins/comments" target="_blank">https://developers.facebook.com/docs/plugins/comments</a></p>
<ul>
<li>
<p>create an App</p>
</li>
<li>
<p>copy and paste the generated code like in the example bleow</p>
</li>
<li>
<p>the <strong>SDK</strong> inclusion script</p>
</li>
<li>
<p>the <strong>.fb-comments DIV</strong> between an <strong>“if statement”</strong> so it becomes simple to disable the comments when needed (globally or in a choosen page)</p>
</li>
</ul>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="c"><!-- Begin include facebook JS SDK --></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"fb-root"</span><span class="nt">></div></span>
<span class="nt"><script></span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">,</span> <span class="nx">s</span><span class="p">,</span> <span class="nx">id</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">js</span><span class="p">,</span> <span class="nx">fjs</span> <span class="o">=</span> <span class="nx">d</span><span class="p">.</span><span class="nx">getElementsByTagName</span><span class="p">(</span><span class="nx">s</span><span class="p">)[</span><span class="mi">0</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="nx">id</span><span class="p">))</span> <span class="k">return</span><span class="p">;</span>
<span class="nx">js</span> <span class="o">=</span> <span class="nx">d</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="nx">s</span><span class="p">);</span> <span class="nx">js</span><span class="p">.</span><span class="nx">id</span> <span class="o">=</span> <span class="nx">id</span><span class="p">;</span>
<span class="nx">js</span><span class="p">.</span><span class="nx">src</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">//connect.facebook.net/fr_FR/sdk.js#xfbml=1&appId=YourAppID&version=v2.0</span><span class="dl">"</span><span class="p">;</span>
<span class="nx">fjs</span><span class="p">.</span><span class="nx">parentNode</span><span class="p">.</span><span class="nx">insertBefore</span><span class="p">(</span><span class="nx">js</span><span class="p">,</span> <span class="nx">fjs</span><span class="p">);</span>
<span class="p">}(</span><span class="nb">document</span><span class="p">,</span> <span class="dl">'</span><span class="s1">script</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">facebook-jssdk</span><span class="dl">'</span><span class="p">));</span>
<span class="nt"></script></span>
<span class="c"><!-- End include facebook JS SDK --></span>
<span class="c"><!-- Begin Comments DIV --></span>
{% if site.comments and if page.comments != false %}
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"fb-comments"</span> <span class="na">data-href=</span><span class="s">" {{ site.url } {{page.url }}"</span> <span class="na">data-numposts=</span><span class="s">"5"</span> <span class="na">data-colorscheme=</span><span class="s">"light"</span><span class="nt">></div></span>
{% endif %}
<span class="c"><!-- End Comments DIV --></span></code></pre></figure>
<p>Add a line in your <strong>_config.yml</strong>:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">comments</span><span class="pi">:</span> <span class="s">yes</span></code></pre></figure>
<p>And Now your are ready to get comments on your articles</p>
<hr />
<p>We can also use Google plus comments but this is not a feature released from Google as a service to use it’s just some code lines stolen from blogger service that embed a comments widget that you can use in your page.</p>
<p>Google can at any time change it’s code since it’s not an official release.</p>
<p>Here’s the code that you need:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html">{% if site.comments and if page.comments != false %}
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://apis.google.com/js/plusone.js"</span><span class="nt">></span>
<span class="nt"></script></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"g-comments"</span>
<span class="na">data-href=</span><span class="s">"https://hanfi.github.io//blog/social-integration-in-jekyll"</span>
<span class="na">data-width=</span><span class="s">"642"</span>
<span class="na">data-first_party_property=</span><span class="s">"BLOGGER"</span>
<span class="na">data-view_type=</span><span class="s">"FILTERED_POSTMOD"</span><span class="nt">></span>
<span class="nt"></div></span>
{% endif %}</code></pre></figure>
<p>the width being hard coded i hate those widgets that are not responsive.</p>
<hr />
<p>Now we are going to add some sharing buttons (facebook, google+, Twitter)</p>
<p>Here are the URLs to get the code to include:</p>
<ul>
<li>
<p><a href="https://about.twitter.com/fr/resources/buttons" target="_blank">https://about.twitter.com/fr/resources/buttons</a> for twitter button</p>
</li>
<li>
<p><a href="https://developers.facebook.com/docs/plugins/like-button" target="_blank">https://developers.facebook.com/docs/plugins/like-button</a> for facebook like/share button</p>
</li>
</ul>
<figure class="highlight"><pre><code class="language-html" data-lang="html">{% if site.share or page.share != false %}
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"social_share"</span><span class="nt">></span>
<span class="nt"><span></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"fb-like"</span> <span class="na">data-href=</span><span class="s">" {{ site.url } {{page.url }}"</span> <span class="na">data-layout=</span><span class="s">"button_count"</span> <span class="na">data-action=</span><span class="s">"like"</span> <span class="na">data-show-faces=</span><span class="s">"false"</span> <span class="na">data-share=</span><span class="s">"true"</span> <span class="na">style=</span><span class="s">"padding-bottom:30px"</span><span class="nt">></div></span>
<span class="nt"></span></span>
<span class="nt"><span></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://apis.google.com/js/platform.js"</span> <span class="na">async</span> <span class="na">defer</span><span class="nt">></span>
<span class="p">{</span><span class="nl">lang</span><span class="p">:</span> <span class="dl">'</span><span class="s1">fr</span><span class="dl">'</span><span class="p">}</span>
<span class="nt"></script></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"g-plusone"</span> <span class="na">data-size=</span><span class="s">"medium"</span><span class="nt">></div></span>
<span class="nt"></span></span>
<span class="nt"><span></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"https://twitter.com/share"</span> <span class="na">class=</span><span class="s">"twitter-share-button"</span><span class="nt">></span>Tweet<span class="nt"></a></span>
<span class="nt"><script></span><span class="o">!</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">,</span><span class="nx">s</span><span class="p">,</span><span class="nx">id</span><span class="p">){</span><span class="kd">var</span> <span class="nx">js</span><span class="p">,</span><span class="nx">fjs</span><span class="o">=</span><span class="nx">d</span><span class="p">.</span><span class="nx">getElementsByTagName</span><span class="p">(</span><span class="nx">s</span><span class="p">)[</span><span class="mi">0</span><span class="p">],</span><span class="nx">p</span><span class="o">=</span><span class="sr">/^http:/</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">location</span><span class="p">)?</span><span class="dl">'</span><span class="s1">http</span><span class="dl">'</span><span class="p">:</span><span class="dl">'</span><span class="s1">https</span><span class="dl">'</span><span class="p">;</span><span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="nx">d</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="nx">id</span><span class="p">)){</span><span class="nx">js</span><span class="o">=</span><span class="nx">d</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="nx">s</span><span class="p">);</span><span class="nx">js</span><span class="p">.</span><span class="nx">id</span><span class="o">=</span><span class="nx">id</span><span class="p">;</span><span class="nx">js</span><span class="p">.</span><span class="nx">src</span><span class="o">=</span><span class="nx">p</span><span class="o">+</span><span class="dl">'</span><span class="s1">://platform.twitter.com/widgets.js</span><span class="dl">'</span><span class="p">;</span><span class="nx">fjs</span><span class="p">.</span><span class="nx">parentNode</span><span class="p">.</span><span class="nx">insertBefore</span><span class="p">(</span><span class="nx">js</span><span class="p">,</span><span class="nx">fjs</span><span class="p">);}}(</span><span class="nb">document</span><span class="p">,</span> <span class="dl">'</span><span class="s1">script</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">twitter-wjs</span><span class="dl">'</span><span class="p">);</span><span class="nt"></script></span>
<span class="nt"></span></span>
<span class="nt"></div></span>
{% endif %}</code></pre></figure>
<p>And dont forget to add this to your <strong>_config.yml</strong> :</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">share</span><span class="pi">:</span> <span class="s">yes</span></code></pre></figure>
<p>And that’s it … <strong>Now share and Comment</strong></p>
Blogging made easy with github.io and jekyll
2014-08-22T00:00:00+00:00
https://hanfi.github.io//blog/githubio-and-jekyll
<p>In this article i will briefly explain how to crate a YourUserName.github.io page and use jekyll to blog.</p>
<p>For more informations <strong>RTFM</strong> : <a href="https://jekyllrb.com/docs/home/" target="_blank">jekyll Doc</a></p>
<p>First of all we will start by creating our github Repository:</p>
<p><a href="https://github.com/" target="_blank">You need a Github Account</a> if you don’t already have one.</p>
<p>Create a new repository called “YourUsername.github.io”</p>
<p>Do a first init/add/commit/push into your remote repo as it’s shown in the new Repository window.</p>
<p>Create your Jekyll blog in your local project folder by following the <a href="https://jekyllrb.com/docs/quickstart/">Jekyll Quickstart</a>.</p>
<p>git add/commit/push and that’s it … your Blog is online at YourGithubUsername.github.com (wait a minute maybe for github to make your site available)</p>
<p>If you need help for Markdown, check this <a href="https://daringfireball.net/projects/markdown/syntax">https://daringfireball.net/projects/markdown/syntax</a>.</p>
<h1 id="and-now-go-blogging-">AND NOW GO BLOGGING !!!</h1>