NskmNever stray from the way.Zola2026-05-02T00:00:00+00:00https://nskm.xyz/atom.xmlITIL 4 Foundations2026-05-02T00:00:00+00:002026-05-02T00:00:00+00:00
Unknown
https://nskm.xyz/posts/itil-4/<style>
.it { border-collapse: collapse; width: 100%; margin: 1.2em 0; font-size: 0.88em; }
.it th, .it td { border: 1px solid #30363d; padding: 8px 13px; text-align: left; vertical-align: top; line-height: 1.5; }
.it thead th { background: #161b22; color: #388bfd; font-weight: 600; letter-spacing: 0.02em; }
.it tbody tr:nth-child(even) td { background: #0d111799; }
</style>
<img style="display: block; margin: 0 auto; width: 500px" src="/images/that-word.png" alt="what?" title="what?"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the author, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, or affiliated groups.</p>
</li>
<li>
<p>If you are a colleague and recognize the situation described below, remember: I'm critiquing the system, not the people.</p>
</li>
<li>
<p>Render unto Caesar what is Caesar's: Claude helped me find up-to-date documentation links and helped beautify all the diagrams used (and not used) in this article.</p>
</li>
<li>
<p>This article has been proofread by <a href="https://www.linkedin.com/in/mameastougassama/">Mame Astou</a> and <a href="https://www.linkedin.com/in/mohamed-abdoul-karim-diop-76250b127/">Abdoul Karim</a>. Thank you both.
<br>
<br></p>
</li>
</ol>
<h2 id="hook">Hook</h2>
<p>I tend to stay in my lane. Focus on what I understand, the perimeter I actually control. Everything beyond my code, my systems, my terminal <em>(the hierarchy, management, all the people I work with who don't write programs)</em> has always been a blur to me. Vague, formless, abstract. Sometimes genuinely incomprehensible. Not because I think those people are wrong, but because whatever they do has always felt like noise I couldn't parse.</p>
<p>I'll grudgingly admit, though: since joining this <a href="https://institutpasteurdakar.sn/">biomedical research center</a>, everything that happens <em>outside</em> my direct work has become the most interesting part of the job. Humans are fascinating creatures to observe from a safe distance. Maybe it's because my actual technical work is no longer the hardest thing on my plate. Or maybe because what happens in the corridors and meeting rooms has a direct, outsized impact on my work, on my <a href="#">freedom</a> to do that work the way I'd want to.</p>
<p>The hardest thing is collaborating with many other people, exchanging information in a way that everyone actually understands each other. Not just hears: understands. The hardest thing is listening, really listening to what others have to say, deeply, absorbing and retaining what they meant. Then responding, expressing yourself in a way that gets through, in a way that convinces. Making the right compromises, the right calls, and moving forward together, making sure everyone gets something of what they came for.</p>
<hr />
<p>Last Tuesday, I was enrolled in an <strong>ITIL 4 Foundations</strong> training organized by my hierarchy. The stated goals were clear: <em>garantir la disponibilité des systèmes</em> (ensure system availability), <em>structurer la gestion des incidents</em> (structure incident management), and <em>parler le même langage</em> (speak the same language). Perfectly reasonable goals.</p>
<p>In practice: sitting in a room while someone talks through concept after concept, deploying words and definitions at high velocity, is its own endurance sport. It's made harder by the fact that some concepts I thought I already knew get redefined with vocabulary I would never have chosen — and some familiar words get used in ways that are subtly but persistently disorienting.</p>
<p>That said, I have to be honest: <strong>ITIL 4 is not the bureaucratic nightmare its reputation suggests</strong>. If anything, I realized I'd already been practicing most of these concepts; informally, imperfectly, without knowing there was a name for them. Done right, this is a coherent framework that actually acknowledges modern realities: Agile, DevOps, SRE, Lean.</p>
<p>Since I took notes anyway, might as well turn them into something useful. I hope you'll learn something. <a href="https://www.youtube.com/watch?v=mbf9aLD8E8M">Music</a> \o/</p>
<h2 id="table-of-contents">Table of contents</h2>
<ol>
<li><a href="https://nskm.xyz/posts/itil-4/#what-is-itil-4">What is itil 4?</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#why-should-your-organization-care">Why should your organization care?</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#the-vocabulary">The vocabulary</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#pestle-the-external-forces-you-cannot-control">Pestle: external forces</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#the-big-picture-service-value-system-svs">The service value system (svs)</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#inside-the-svs-the-service-value-chain-svc">The service value chain (svc)</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#the-4-dimensions">The 4 dimensions</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#the-7-guiding-principles">The 7 guiding principles</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#the-34-management-practices">The 34 practices</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#incident-management">Incident management</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#change-enablement">Change enablement</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#continual-improvement">Continual improvement</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#governance-and-risk-management">Governance and risk management</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#itil-4-agile-devops-lean">ITIL 4 ↔ agile, devops, lean</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#case-study">Case study</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#tooling">Tooling</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#itil-v5">ITIL v5</a></li>
<li><a href="https://nskm.xyz/posts/itil-4/#conclusion">Conclusion</a></li>
</ol>
<p><br><br></p>
<h2 id="what-is-itil-4">What is ITIL 4?</h2>
<p><strong><a href="https://www.axelos.com/certifications/itil-service-management">ITIL (Information Technology Infrastructure Library)</a></strong> is a framework for IT Service Management (ITSM). It provides guidance on how organizations can use IT as a tool to facilitate business transformation and growth.</p>
<p>Key facts:</p>
<ul>
<li><strong>Origin</strong>: Developed in the 1980s by the UK government's CCTA (Central Computer and Telecommunications Agency) to standardize IT practices across public services.</li>
<li><strong>ITIL 4</strong> (released <strong>2019</strong>): The current version. A major rethink compared to ITIL v3, designed to integrate with modern approaches: Agile, DevOps, Lean, SRE.</li>
<li><strong>Owner</strong>: Currently maintained by <strong><a href="https://www.peoplecert.org/">PeopleCert</a></strong>, which acquired AXELOS (the previous owner) in 2021.</li>
<li><strong>Who uses it?</strong> Organizations of all sizes, across industries. Particularly common in banking, telecom, healthcare, public sector, and large IT departments.</li>
</ul>
<p>One thing ITIL 4 gets right that its predecessors didn't: it explicitly acknowledges that <strong>you don't need to implement all 34 practices</strong>. You pick what fits. This was not how earlier versions were often sold and it caused a lot of unnecessary process theater in organizations that implemented ITIL like a checkbox exercise.</p>
<p><br><br></p>
<h2 id="why-should-your-organization-care">Why should your organization care?</h2>
<p>During the training, an interesting observation was made about three organizational thresholds where ITIL practices start to become relevant:</p>
<table class="it">
<thead><tr><th>Threshold</th><th>Pain Point</th><th>What Starts Breaking</th></tr></thead>
<tbody>
<tr><td><strong>≥ 5 people</strong></td><td>Memory loss</td><td>Knowledge lives in people's heads, not in runbooks</td></tr>
<tr><td><strong>≥ 10 people</strong></td><td>Complexity</td><td>Coordination overhead grows faster than the team</td></tr>
<tr><td><strong>> 10 people</strong></td><td>Compliance</td><td>Audits, regulations, and reporting requirements appear</td></tr>
</tbody>
</table>
<p>The practical implication: <strong>you don't need all of ITIL</strong>. A 7-person startup building an internal tool doesn't need a Change Advisory Board. A bank with 200 IT staff probably does. ITIL 4's design philosophy is, understand the principles, apply the practices that solve your actual problems, and leave the rest on the shelf.</p>
<p><br><br></p>
<h2 id="the-vocabulary">The vocabulary</h2>
<p>Before we can discuss the framework, we need to agree on definitions. ITIL has precise meanings for words we use casually.</p>
<h3 id="who-s-involved">Who's involved?</h3>
<table class="it">
<thead><tr><th>Term</th><th>Definition</th></tr></thead>
<tbody>
<tr><td><strong>Organization</strong></td><td>A person or group with its own functions, responsibilities, and authorities</td></tr>
<tr><td><strong>Stakeholder</strong></td><td>Any person with an interest in an organization, product, or service</td></tr>
<tr><td><strong>Service Provider</strong></td><td>An organization that delivers services to consumers</td></tr>
<tr><td><strong>Service Consumer</strong></td><td>An organization or individual that receives services</td></tr>
<tr><td><strong>Customer</strong></td><td>Defines requirements and owns the outcomes</td></tr>
<tr><td><strong>User</strong></td><td>The person who uses the service day-to-day</td></tr>
<tr><td><strong>Sponsor</strong></td><td>The person who authorizes the budget</td></tr>
</tbody>
</table>
<p>In practice: the <strong>customer</strong> decides <em>what</em> they need, the <strong>user</strong> interacts with it daily, and the <strong>sponsor</strong> pays for it.</p>
<h3 id="what-are-we-delivering">What are we delivering?</h3>
<table class="it">
<thead><tr><th>Term</th><th>Definition</th></tr></thead>
<tbody>
<tr><td><strong>Resource</strong></td><td>Any component used to deliver value: people, infrastructure, capital, information</td></tr>
<tr><td><strong>Product</strong></td><td>A configuration of resources designed to offer value to a consumer</td></tr>
<tr><td><strong>Service</strong></td><td>A means of enabling value co-creation by facilitating outcomes that customers want</td></tr>
<tr><td><strong>Service Offering</strong></td><td>A formal description of one or more services, designed for a target consumer group</td></tr>
</tbody>
</table>
<h3 id="the-value-equation">The value equation</h3>
<p>This is arguably the most important concept in ITIL v4:</p>
<blockquote>
<p><strong>Value = Utility × Warranty</strong></p>
</blockquote>
<ul>
<li><strong>Utility</strong>: What the service does — <em>fit for purpose</em>. Does it support the customer's performance or remove a constraint?</li>
<li><strong>Warranty</strong>: How reliably it does it — <em>fit for use</em>. Is it available when needed? Secure? Capable of handling the load?</li>
</ul>
<p>A service that does the right thing but crashes every Monday morning has no value. A service that's always available but solves the wrong problem also has no value. We need both, because a zero on either side collapses the product.</p>
<p>And value is <strong>subjective, contextual, and co-created</strong>. The provider alone cannot create value, it's always produced in interaction with the customer. This shifts the conversation from <em>"we delivered the system"</em> to <em>"did the customer actually get what they needed?"</em></p>
<div class="itil4-actors-wrapper">
<style>
.itil4-actors-wrapper {
--bg: #0d1117;
--bg2: #161b22;
--bg3: #21262d;
--border: #30363d;
--text: #e6edf3;
--muted: #8b949e;
--blue: #388bfd;
--blue-d: #1f6feb;
--green: #3fb950;
--purple: #a371f7;
--orange: #f0883e;
--yellow: #d29922;
font-family: 'IBM Plex Sans', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
background: var(--bg);
color: var(--text);
padding: 28px 20px 20px;
border-radius: 10px;
margin: 30px 0;
}
.itil4-actors-wrapper * { margin: 0; padding: 0; box-sizing: border-box; }
.act-header { text-align: center; margin-bottom: 22px; }
.act-header h2 { font-size: 1.2rem; font-weight: 700; color: var(--blue); text-transform: uppercase; letter-spacing: 0.05em; }
.act-header p { color: var(--muted); font-size: 0.8rem; margin-top: 5px; }
/* ── 3-column main layout ── */
.act-main {
display: grid;
grid-template-columns: 1fr auto 1fr;
gap: 12px;
align-items: center;
}
/* ── lane (provider / consumer) ── */
.act-lane {
border: 1px solid var(--border);
border-radius: 10px;
overflow: hidden;
}
.act-lane-header {
padding: 7px 12px;
font-size: 0.72rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.07em;
text-align: center;
}
.act-lane-header.provider { background: rgba(56,139,253,0.15); color: var(--blue); }
.act-lane-header.consumer { background: rgba(163,113,247,0.15); color: var(--purple); }
.act-lane-body { padding: 10px; display: flex; flex-direction: column; gap: 0; background: var(--bg2); }
/* ── nodes ── */
.act-node {
border: 1px solid var(--border);
border-radius: 7px;
padding: 9px 12px;
cursor: pointer;
transition: border-color 0.15s, transform 0.1s;
background: var(--bg3);
}
.act-node:hover { border-color: var(--blue); transform: translateX(2px); }
.act-node.active { border-color: var(--blue); box-shadow: 0 0 0 2px rgba(56,139,253,0.2); }
.act-node .nd-icon { font-size: 1.1rem; }
.act-node .nd-name { font-size: 0.78rem; font-weight: 700; color: var(--text); margin-top: 3px; }
.act-node .nd-role { font-size: 0.68rem; color: var(--muted); margin-top: 2px; line-height: 1.4; }
/* chain arrow between nodes in a lane */
.act-chain-arrow {
text-align: center;
color: var(--muted);
font-size: 0.9rem;
padding: 2px 0;
line-height: 1;
}
/* ── centre exchange column ── */
.act-exchange {
display: flex;
flex-direction: column;
align-items: center;
gap: 8px;
min-width: 110px;
padding: 0 4px;
}
.act-flow-row {
display: flex;
align-items: center;
gap: 5px;
font-size: 0.7rem;
color: var(--muted);
white-space: nowrap;
}
.act-flow-arrow { font-size: 1rem; }
.act-flow-label { font-size: 0.65rem; color: var(--muted); text-align: center; }
.act-value-box {
border: 1px solid var(--green);
border-radius: 8px;
padding: 9px 10px;
text-align: center;
background: rgba(63,185,80,0.06);
cursor: pointer;
transition: border-color 0.15s;
}
.act-value-box:hover { border-color: #5af07d; }
.act-value-box.active { box-shadow: 0 0 0 2px rgba(63,185,80,0.25); }
.act-value-box .vb-eq { font-size: 0.8rem; font-weight: 700; color: var(--green); }
.act-value-box .vb-sub { font-size: 0.65rem; color: var(--muted); margin-top: 3px; line-height: 1.4; }
.act-cocreate {
font-size: 0.62rem;
color: var(--muted);
text-align: center;
font-style: italic;
padding: 0 4px;
}
/* ── info panel ── */
.act-panel {
margin-top: 16px;
background: var(--bg3);
border: 1px solid var(--border);
border-radius: 8px;
padding: 14px 16px;
font-size: 0.84rem;
min-height: 72px;
}
.act-panel .ap-title { font-weight: 700; color: var(--blue); margin-bottom: 5px; font-size: 0.88rem; }
.act-panel .ap-desc { color: var(--text); line-height: 1.6; }
.act-panel .ap-hint { color: var(--muted); font-style: italic; font-size: 0.78rem; }
.act-panel .ap-tags { display: flex; flex-wrap: wrap; gap: 5px; margin-top: 8px; }
.act-panel .ap-tag { font-size: 0.7rem; background: var(--bg2); border: 1px solid var(--border); border-radius: 4px; padding: 2px 8px; color: var(--orange); }
@media (max-width: 580px) {
.act-main { grid-template-columns: 1fr; }
.act-exchange { flex-direction: row; flex-wrap: wrap; justify-content: center; min-width: unset; }
.act-flow-row { font-size: 0.65rem; }
}
</style>
<div class="act-header">
<h2>Who does what — for whom</h2>
<p>Click any node to see its ITIL 4 definition</p>
</div>
<div class="act-main">
<!-- LEFT: Service Provider lane -->
<div class="act-lane">
<div class="act-lane-header provider">⚙ Service Provider</div>
<div class="act-lane-body">
<div class="act-node" data-act="resource">
<span class="nd-icon">🧱</span>
<div class="nd-name">Resource</div>
<div class="nd-role">Raw input to build with</div>
</div>
<div class="act-chain-arrow">↓ assembles into</div>
<div class="act-node" data-act="product">
<span class="nd-icon">📦</span>
<div class="nd-name">Product</div>
<div class="nd-role">Configured for value delivery</div>
</div>
<div class="act-chain-arrow">↓ packaged as</div>
<div class="act-node" data-act="service">
<span class="nd-icon">⚡</span>
<div class="nd-name">Service</div>
<div class="nd-role">Enables consumer outcomes</div>
</div>
<div class="act-chain-arrow">↓ described in</div>
<div class="act-node" data-act="offering">
<span class="nd-icon">📋</span>
<div class="nd-name">Service Offering</div>
<div class="nd-role">Formal promise to the consumer</div>
</div>
</div>
</div>
<!-- CENTRE: exchange -->
<div class="act-exchange">
<div class="act-flow-row">
<span class="act-flow-arrow" style="color:#388bfd">→</span>
<span class="act-flow-label">delivers</span>
</div>
<div class="act-value-box" data-act="value">
<div class="vb-eq">Utility × Warranty</div>
<div class="vb-sub">= Value</div>
</div>
<div class="act-cocreate">co-created<br>by both sides</div>
<div class="act-flow-row">
<span class="act-flow-label">requires</span>
<span class="act-flow-arrow" style="color:#a371f7">←</span>
</div>
</div>
<!-- RIGHT: Service Consumer lane -->
<div class="act-lane">
<div class="act-lane-header consumer">👥 Service Consumer</div>
<div class="act-lane-body">
<div class="act-node" data-act="sponsor">
<span class="nd-icon">💰</span>
<div class="nd-name">Sponsor</div>
<div class="nd-role">Authorizes the budget</div>
</div>
<div class="act-chain-arrow">↓ funds</div>
<div class="act-node" data-act="customer">
<span class="nd-icon">🎯</span>
<div class="nd-name">Customer</div>
<div class="nd-role">Defines requirements & owns outcomes</div>
</div>
<div class="act-chain-arrow">↓ specifies for</div>
<div class="act-node" data-act="user">
<span class="nd-icon">🖥</span>
<div class="nd-name">User</div>
<div class="nd-role">Uses the service day-to-day</div>
</div>
<div class="act-chain-arrow">↓ feeds back to</div>
<div class="act-node" data-act="stakeholder">
<span class="nd-icon">🌐</span>
<div class="nd-name">Stakeholder</div>
<div class="nd-role">Any party with an interest</div>
</div>
</div>
</div>
</div><!-- /act-main -->
<div class="act-panel" id="act-panel">
<div class="ap-hint">← Click any node to see its definition</div>
</div>
<script>
(function () {
const defs = {
resource: {
title: "Resource",
desc: "Any component that can be used to deliver value: people, infrastructure, capital, information, or time. Resources are the raw inputs that the service provider assembles into something useful.",
tags: ["People", "Infrastructure", "Capital", "Information"]
},
product: {
title: "Product",
desc: "A configuration of resources designed to offer value to a consumer. A product is what the service provider builds — it may not be directly visible to the consumer, but it's what the service is built on top of.",
tags: ["Configuration of resources", "Provider-facing", "Foundation of a service"]
},
service: {
title: "Service",
desc: "A means of enabling value co-creation by facilitating outcomes that customers want to achieve, without the customer having to manage specific costs and risks. The key insight: value is not delivered, it is co-created.",
tags: ["Enables outcomes", "Removes customer risk", "Co-created with consumer"]
},
offering: {
title: "Service Offering",
desc: "A formal description of one or more services, designed to address the needs of a target consumer group. It's the official 'promise' from provider to consumer — what's included, what's not, and at what terms.",
tags: ["Goods", "Access to resources", "Service actions", "Terms & conditions"]
},
sponsor: {
title: "Sponsor",
desc: "The person who authorizes the budget for the consumption of a service. The sponsor may be the customer, or a separate person (e.g., a CFO who approves a service contract that the IT team actually uses).",
tags: ["Authorizes budget", "May differ from Customer", "Financial accountability"]
},
customer: {
title: "Customer",
desc: "The person who defines the requirements for a service and takes responsibility for the outcomes of service consumption. The customer decides what is needed and judges whether the service delivered value.",
tags: ["Defines requirements", "Owns outcomes", "Negotiates SLAs", "May also be Sponsor"]
},
user: {
title: "User",
desc: "The person who uses services in their day-to-day work. Users interact directly with the service and often experience it differently from what the customer specified or the sponsor funded. This gap is why XLAs matter.",
tags: ["Day-to-day usage", "Direct experience", "Source of XLA signals", "May differ from Customer"]
},
stakeholder: {
title: "Stakeholder",
desc: "Any person who has an interest in an organization, product, service, practice, or other entity. Stakeholders include everyone on both sides: providers, consumers, regulators, partners, suppliers, and even the public in some contexts.",
tags: ["Broad category", "Provider side", "Consumer side", "Regulators & partners"]
},
value: {
title: "Value = Utility × Warranty",
desc: "Value is subjective, contextual, and co-created — it cannot be delivered unilaterally by the provider. Utility is fitness for purpose (does the service do what the customer needs?). Warranty is fitness for use (is it available, secure, and reliable enough?). A zero on either factor collapses the value entirely.",
tags: ["Utility = fit for purpose", "Warranty = fit for use", "Co-created", "Subjective & contextual"]
}
};
const panel = document.getElementById('act-panel');
document.querySelectorAll('.itil4-actors-wrapper [data-act]').forEach(el => {
el.addEventListener('click', function () {
const key = this.dataset.act;
const d = defs[key];
if (!d) return;
document.querySelectorAll('.itil4-actors-wrapper [data-act]').forEach(e => e.classList.remove('active'));
this.classList.add('active');
const tags = d.tags.map(t => `<span class="ap-tag">${t}</span>`).join('');
panel.innerHTML = `
<div class="ap-title">${d.title}</div>
<div class="ap-desc">${d.desc}</div>
<div class="ap-tags">${tags}</div>
`;
});
});
})();
</script>
</div>
<p><br><br></p>
<h2 id="pestle-the-external-forces-you-cannot-control">PESTLE: The external forces you cannot control</h2>
<p>ITIL v4 acknowledges that organizations don't operate in a vacuum. <strong>PESTLE analysis</strong> provides a structured lens for scanning the external environment:</p>
<table class="it">
<thead><tr><th>Factor</th><th>What It Covers</th><th>IT Relevance</th></tr></thead>
<tbody>
<tr><td><strong>Political</strong></td><td>Government policies, regulations</td><td>Data sovereignty laws, public sector mandates</td></tr>
<tr><td><strong>Economic</strong></td><td>Budget cycles, cost pressures</td><td>Cloud cost optimization, license negotiation</td></tr>
<tr><td><strong>Social</strong></td><td>User expectations, workforce trends</td><td>Remote work adoption, digital literacy gaps</td></tr>
<tr><td><strong>Technological</strong></td><td>Emerging tech, vendor landscape</td><td>AI integration, cloud migration, security threats</td></tr>
<tr><td><strong>Legal</strong></td><td>Compliance requirements</td><td>GDPR, HIPAA, sector-specific regulations</td></tr>
<tr><td><strong>Environmental</strong></td><td>Sustainability, green IT</td><td>Data center energy, hardware lifecycle</td></tr>
</tbody>
</table>
<img style="display: block; margin: 0 auto; width: 300px" src="/images/pestle.png" alt="pestle" title="pestle"/>
<p><a href="https://www.cipd.org/en/knowledge/factsheets/pestle-analysis-factsheet/">PESTLE</a> is not unique to ITIL, it's a general strategic analysis tool. ITIL v4 borrows it because service management decisions are always made inside an organizational and environmental context. We never want to build a technically excellent system that doesn't survive its first compliance audit or budget cut.</p>
<p><br><br></p>
<h2 id="the-big-picture-service-value-system-svs">The big picture: Service Value System (SVS)</h2>
<p>The <strong>Service Value System (SVS)</strong> is ITIL v4's model for how an organization creates value. Think of it as the engine diagram for your IT service operations.</p>
<p>It takes <strong>opportunities and demands</strong> as input, runs them through a system of governance, practices, and activities, and outputs <strong>value</strong> for all stakeholders.</p>
<p>The SVS is composed of five components:</p>
<ol>
<li><strong>Guiding Principles</strong> — universal recommendations that apply to all decisions</li>
<li><strong>Governance</strong> — the framework for directing and controlling the organization</li>
<li><strong>Service Value Chain (SVC)</strong> — the set of interconnected activities that turn demand into value</li>
<li><strong>Management Practices</strong> — the 34 practices that provide organizational capability</li>
<li><strong>Continual Improvement</strong> — the persistent drive to improve everything</li>
</ol>
<div class="itil4-svs-wrapper">
<style>
.itil4-svs-wrapper {
--bg: #0d1117;
--bg2: #161b22;
--bg3: #21262d;
--border: #30363d;
--text: #e6edf3;
--muted: #8b949e;
--blue: #388bfd;
--blue-dim: #1f6feb;
--green: #3fb950;
--purple: #a371f7;
--orange: #f0883e;
--yellow: #d29922;
--red: #f85149;
font-family: 'IBM Plex Sans', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
background: var(--bg);
color: var(--text);
padding: 32px 20px;
border-radius: 10px;
margin: 30px 0;
}
.itil4-svs-wrapper * { margin: 0; padding: 0; box-sizing: border-box; }
.svs-title-block {
text-align: center;
margin-bottom: 28px;
}
.svs-title-block h2 {
font-size: 1.4rem;
font-weight: 700;
color: var(--blue);
letter-spacing: 0.05em;
text-transform: uppercase;
}
.svs-title-block p {
color: var(--muted);
font-size: 0.85rem;
margin-top: 6px;
}
.svs-main-layout {
display: flex;
align-items: stretch;
gap: 12px;
}
/* --- Input/Output columns --- */
.svs-io {
display: flex;
flex-direction: column;
justify-content: center;
align-items: center;
gap: 16px;
min-width: 90px;
}
.svs-io-item {
background: var(--bg3);
border: 1px solid var(--border);
border-radius: 8px;
padding: 12px 10px;
text-align: center;
font-size: 0.78rem;
width: 90px;
}
.svs-io-item .io-icon { font-size: 1.4rem; display: block; margin-bottom: 4px; }
.svs-io-item .io-label { color: var(--muted); font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.04em; }
.svs-io-item .io-name { color: var(--text); font-weight: 600; margin-top: 2px; }
.svs-arrow {
color: var(--blue);
font-size: 1.6rem;
display: flex;
align-items: center;
justify-content: center;
padding: 0 4px;
}
/* --- SVS box --- */
.svs-box {
flex: 1;
border: 2px solid var(--blue-dim);
border-radius: 12px;
padding: 16px;
position: relative;
background: var(--bg2);
}
.svs-guiding-label, .svs-ci-label {
text-align: center;
font-size: 0.72rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.08em;
padding: 5px 0;
border-radius: 4px;
margin-bottom: 10px;
}
.svs-guiding-label {
color: var(--purple);
border: 1px dashed var(--purple);
opacity: 0.85;
}
.svs-ci-label {
color: var(--green);
border: 1px dashed var(--green);
margin-top: 10px;
margin-bottom: 0;
opacity: 0.85;
}
.svs-inner-grid {
display: grid;
grid-template-columns: 1fr 2fr;
gap: 10px;
margin: 10px 0;
}
.svs-component {
border-radius: 8px;
padding: 12px;
cursor: pointer;
transition: border-color 0.2s, transform 0.1s;
border: 1px solid var(--border);
}
.svs-component:hover { border-color: var(--blue); transform: translateY(-1px); }
.svs-component.active { border-color: var(--blue); box-shadow: 0 0 0 2px rgba(56,139,253,0.25); }
.svs-component .comp-icon { font-size: 1.3rem; margin-bottom: 5px; }
.svs-component .comp-name {
font-size: 0.8rem;
font-weight: 700;
color: var(--text);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.svs-component .comp-sub {
font-size: 0.72rem;
color: var(--muted);
margin-top: 4px;
line-height: 1.4;
}
.comp-governance { background: rgba(163,113,247,0.08); }
.comp-svc { background: rgba(56,139,253,0.08); grid-row: 1 / 3; }
.comp-practices { background: rgba(240,136,62,0.08); }
.svc-activities {
display: flex;
flex-wrap: wrap;
gap: 4px;
margin-top: 8px;
}
.svc-act {
font-size: 0.68rem;
background: var(--bg3);
border: 1px solid var(--border);
border-radius: 4px;
padding: 2px 6px;
color: var(--blue);
font-weight: 500;
}
/* --- Info panel --- */
.svs-info-panel {
margin-top: 20px;
background: var(--bg3);
border: 1px solid var(--border);
border-radius: 8px;
padding: 16px;
font-size: 0.85rem;
min-height: 80px;
transition: all 0.2s;
}
.svs-info-panel .panel-title {
font-weight: 700;
color: var(--blue);
font-size: 0.9rem;
margin-bottom: 6px;
}
.svs-info-panel .panel-desc { color: var(--text); line-height: 1.6; }
.svs-info-panel .panel-hint {
color: var(--muted);
font-size: 0.78rem;
font-style: italic;
}
.svs-info-panel .panel-items {
display: flex;
flex-wrap: wrap;
gap: 6px;
margin-top: 8px;
}
.svs-info-panel .panel-tag {
font-size: 0.72rem;
background: var(--bg2);
border: 1px solid var(--border);
border-radius: 4px;
padding: 2px 8px;
color: var(--text);
}
.svs-legend {
display: flex;
flex-wrap: wrap;
gap: 12px;
margin-top: 16px;
font-size: 0.75rem;
color: var(--muted);
justify-content: center;
}
.svs-legend-item { display: flex; align-items: center; gap: 5px; }
.svs-legend-dot {
width: 8px; height: 8px;
border-radius: 50%;
display: inline-block;
}
@media (max-width: 600px) {
.svs-main-layout { flex-direction: column; }
.svs-arrow { transform: rotate(90deg); }
.svs-inner-grid { grid-template-columns: 1fr; }
.comp-svc { grid-row: auto; }
}
</style>
<div class="svs-title-block">
<h2>ITIL 4 — Service Value System (SVS)</h2>
<p>Click any component to learn more</p>
</div>
<div class="svs-main-layout">
<!-- Inputs -->
<div class="svs-io">
<div class="svs-io-item">
<span class="io-icon">💡</span>
<div class="io-label">Input</div>
<div class="io-name">Opportunity</div>
</div>
<div class="svs-io-item">
<span class="io-icon">📋</span>
<div class="io-label">Input</div>
<div class="io-name">Demand</div>
</div>
</div>
<div class="svs-arrow">→</div>
<!-- SVS Box -->
<div class="svs-box">
<div class="svs-guiding-label" data-svs="guiding" style="cursor:pointer">
✦ Guiding Principles — universally applicable ✦
</div>
<div class="svs-inner-grid">
<div class="svs-component comp-governance" data-svs="governance">
<div class="comp-icon">🏛</div>
<div class="comp-name">Governance</div>
<div class="comp-sub">Direct · Evaluate · Monitor</div>
</div>
<div class="svs-component comp-svc" data-svs="svc">
<div class="comp-icon">⛓</div>
<div class="comp-name">Service Value Chain</div>
<div class="comp-sub">6 interconnected activities that convert demand into value</div>
<div class="svc-activities">
<span class="svc-act">Plan</span>
<span class="svc-act">Engage</span>
<span class="svc-act">Design & Transition</span>
<span class="svc-act">Obtain / Build</span>
<span class="svc-act">Deliver & Support</span>
<span class="svc-act">Improve</span>
</div>
</div>
<div class="svs-component comp-practices" data-svs="practices">
<div class="comp-icon">🔧</div>
<div class="comp-name">34 Practices</div>
<div class="comp-sub">General (14) · Service (17) · Technical (3)</div>
</div>
</div>
<div class="svs-ci-label" data-svs="ci" style="cursor:pointer">
↻ Continual Improvement — embedded at every level ↺
</div>
</div>
<div class="svs-arrow">→</div>
<!-- Output -->
<div class="svs-io">
<div class="svs-io-item" style="width:90px">
<span class="io-icon">✨</span>
<div class="io-label">Output</div>
<div class="io-name">Value for all stakeholders</div>
</div>
</div>
</div>
<!-- Info panel -->
<div class="svs-info-panel" id="svs-panel">
<div class="panel-hint">← Click a component above to see its role in the SVS</div>
</div>
<div class="svs-legend">
<div class="svs-legend-item"><span class="svs-legend-dot" style="background:#a371f7"></span> Guiding Principles</div>
<div class="svs-legend-item"><span class="svs-legend-dot" style="background:#a371f7; opacity:0.5"></span> Governance</div>
<div class="svs-legend-item"><span class="svs-legend-dot" style="background:#388bfd"></span> Service Value Chain</div>
<div class="svs-legend-item"><span class="svs-legend-dot" style="background:#f0883e"></span> 34 Practices</div>
<div class="svs-legend-item"><span class="svs-legend-dot" style="background:#3fb950"></span> Continual Improvement</div>
</div>
<script>
(function() {
const info = {
guiding: {
title: "7 Guiding Principles",
desc: "Universal recommendations that guide organizations in all circumstances. Unlike practices or processes, they apply regardless of what you're doing or where you are in the SVS. They cannot be overridden by management or policy.",
items: ["Focus on value", "Start where you are", "Progress iteratively with feedback", "Collaborate and promote visibility", "Think and work holistically", "Keep it simple and practical", "Optimize and automate"]
},
governance: {
title: "Governance",
desc: "The framework by which an organization is directed and controlled. In the SVS, governance sits above everything else — it sets the direction and ensures accountability. Three core functions:",
items: ["Direct — set strategy, objectives, policies", "Evaluate — assess performance, risk, compliance", "Monitor — track adherence and verify outcomes"]
},
svc: {
title: "Service Value Chain (SVC)",
desc: "The operational model at the heart of the SVS. Six activities that can be combined in different ways to create value streams — the specific sequences of steps for a given product or service. It is NOT linear: each value stream activates a different combination of activities.",
items: ["Plan — shared understanding of direction", "Engage — stakeholder needs and relationships", "Design & Transition — meet stakeholder expectations", "Obtain/Build — components available when needed", "Deliver & Support — services per agreed specs", "Improve — continual improvement across all services"]
},
practices: {
title: "34 Management Practices",
desc: "Sets of organizational resources designed for performing work or accomplishing an objective. Organized into three categories. You do not need to implement all 34 — adopt what solves your actual problems.",
items: ["General Management (14): strategy, risk, knowledge, continual improvement...", "Service Management (17): incident, change, service desk, problem...", "Technical Management (3): deployment, infrastructure, software dev"]
},
ci: {
title: "Continual Improvement",
desc: "A recurring practice embedded across the entire SVS — not just a step at the end. Every component, every practice, every activity should be subject to improvement. The Continual Improvement Register (CIR) tracks improvement ideas and their status.",
items: ["7-step model: vision → baseline → target → plan → act → measure → sustain", "3 axes: strategic, tactical, operational", "Tools: OKR, KPI, SLA, XLA, CIR"]
}
};
const panel = document.getElementById('svs-panel');
document.querySelectorAll('.itil4-svs-wrapper [data-svs]').forEach(el => {
el.addEventListener('click', function() {
const key = this.dataset.svs;
const d = info[key];
if (!d) return;
document.querySelectorAll('.itil4-svs-wrapper [data-svs]').forEach(e => e.classList.remove('active'));
this.classList.add('active');
const tagsHtml = d.items.map(i => `<span class="panel-tag">${i}</span>`).join('');
panel.innerHTML = `
<div class="panel-title">${d.title}</div>
<div class="panel-desc">${d.desc}</div>
<div class="panel-items">${tagsHtml}</div>
`;
});
});
})();
</script>
</div>
<p>NOTA BENE: <strong>Guiding Principles</strong> and <strong>Continual Improvement</strong> are not steps in a sequence. They're a mindset that applies to all activities, at all times. You don't "do continual improvement" after you're done. You do it while you're doing everything else.</p>
<p><br><br></p>
<h2 id="inside-the-svs-the-service-value-chain-svc">Inside the SVS: the Service Value Chain (SVC)</h2>
<p>The <strong>Service Value Chain</strong> is the operational model at the core of the SVS. It defines 6 activities that can be combined in different ways. The combination for a given service or request is called a <strong>value stream</strong>.</p>
<table class="it">
<thead><tr><th>Activity</th><th>Purpose</th></tr></thead>
<tbody>
<tr><td><strong>Plan</strong></td><td>Shared understanding of vision, direction, and current state</td></tr>
<tr><td><strong>Engage</strong></td><td>Understand stakeholder needs; maintain transparency and relationships</td></tr>
<tr><td><strong>Design & Transition</strong></td><td>Ensure products and services meet stakeholder expectations</td></tr>
<tr><td><strong>Obtain / Build</strong></td><td>Ensure service components are available when and where needed</td></tr>
<tr><td><strong>Deliver & Support</strong></td><td>Ensure services are delivered and supported per agreed specifications</td></tr>
<tr><td><strong>Improve</strong></td><td>Continual improvement across all products, services, and practices</td></tr>
</tbody>
</table>
<p>The SVC is <strong>not linear</strong>. An incident resolution primarily activates Engage, Deliver & Support, and Improve. A new service launch activates all six. A routine service request might only need Engage and Deliver & Support.</p>
<div class="itil4-svc-wrapper">
<style>
.itil4-svc-wrapper {
--bg: #0d1117;
--bg2: #161b22;
--bg3: #21262d;
--border: #30363d;
--text: #e6edf3;
--muted: #8b949e;
--blue: #388bfd;
--blue-dim: #1f6feb;
--green: #3fb950;
--purple: #a371f7;
--orange: #f0883e;
font-family: 'IBM Plex Sans', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
background: var(--bg);
color: var(--text);
padding: 28px 20px;
border-radius: 10px;
margin: 30px 0;
}
.itil4-svc-wrapper * { margin: 0; padding: 0; box-sizing: border-box; }
.svc-header {
text-align: center;
margin-bottom: 24px;
}
.svc-header h2 {
font-size: 1.3rem;
font-weight: 700;
color: var(--blue);
text-transform: uppercase;
letter-spacing: 0.05em;
}
.svc-header p {
color: var(--muted);
font-size: 0.82rem;
margin-top: 5px;
}
.svc-flow {
display: flex;
flex-wrap: wrap;
align-items: center;
justify-content: center;
gap: 6px;
margin-bottom: 20px;
}
.svc-node {
background: var(--bg2);
border: 1px solid var(--border);
border-radius: 8px;
padding: 10px 14px;
cursor: pointer;
text-align: center;
min-width: 110px;
transition: border-color 0.2s, transform 0.1s, background 0.2s;
}
.svc-node:hover { border-color: var(--blue); transform: translateY(-2px); }
.svc-node.active { border-color: var(--blue); background: rgba(56,139,253,0.1); }
.svc-node .node-icon { font-size: 1.3rem; display: block; margin-bottom: 4px; }
.svc-node .node-name {
font-size: 0.78rem;
font-weight: 700;
color: var(--text);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.svc-node .node-sub {
font-size: 0.68rem;
color: var(--muted);
margin-top: 3px;
}
.svc-connector {
color: var(--blue-dim);
font-size: 1.1rem;
opacity: 0.7;
}
.svc-note {
text-align: center;
font-size: 0.75rem;
color: var(--muted);
margin-bottom: 16px;
font-style: italic;
}
.svc-info-panel {
background: var(--bg3);
border: 1px solid var(--border);
border-radius: 8px;
padding: 16px;
font-size: 0.84rem;
min-height: 70px;
}
.svc-info-panel .panel-title {
font-weight: 700;
color: var(--blue);
margin-bottom: 5px;
}
.svc-info-panel .panel-desc { color: var(--text); line-height: 1.6; }
.svc-info-panel .panel-hint { color: var(--muted); font-style: italic; font-size: 0.78rem; }
.svc-info-panel .panel-examples {
margin-top: 8px;
font-size: 0.78rem;
color: var(--muted);
}
.svc-info-panel .panel-examples span {
background: var(--bg2);
border: 1px solid var(--border);
border-radius: 4px;
padding: 1px 7px;
margin-right: 4px;
display: inline-block;
margin-top: 4px;
color: var(--orange);
}
@media (max-width: 600px) {
.svc-flow { flex-direction: column; }
.svc-connector { transform: rotate(90deg); }
}
</style>
<div class="svc-header">
<h2>Service Value Chain (SVC)</h2>
<p>Six interconnected activities — combined differently for each value stream</p>
</div>
<div class="svc-flow">
<div class="svc-node" data-svc="plan">
<span class="node-icon">🗺</span>
<div class="node-name">Plan</div>
<div class="node-sub">Direction & strategy</div>
</div>
<div class="svc-connector">↔</div>
<div class="svc-node" data-svc="engage">
<span class="node-icon">🤝</span>
<div class="node-name">Engage</div>
<div class="node-sub">Stakeholders & needs</div>
</div>
<div class="svc-connector">↔</div>
<div class="svc-node" data-svc="design">
<span class="node-icon">📐</span>
<div class="node-name">Design & Transition</div>
<div class="node-sub">Meet expectations</div>
</div>
<div class="svc-connector">↔</div>
<div class="svc-node" data-svc="build">
<span class="node-icon">🔨</span>
<div class="node-name">Obtain / Build</div>
<div class="node-sub">Components ready</div>
</div>
<div class="svc-connector">↔</div>
<div class="svc-node" data-svc="deliver">
<span class="node-icon">🚀</span>
<div class="node-name">Deliver & Support</div>
<div class="node-sub">Run & resolve</div>
</div>
<div class="svc-connector">↔</div>
<div class="svc-node" data-svc="improve">
<span class="node-icon">📈</span>
<div class="node-name">Improve</div>
<div class="node-sub">Always, for everything</div>
</div>
</div>
<div class="svc-note">↔ Activities are non-linear: each service interaction combines them differently</div>
<div class="svc-info-panel" id="svc-panel">
<div class="panel-hint">← Click an activity above to see what it covers</div>
</div>
<script>
(function() {
const info = {
plan: {
title: "Plan",
desc: "Ensures a shared understanding of vision, current status, and direction for all four dimensions. Produces plans, policies, portfolios, and architectures used by all other activities.",
examples: ["Capacity planning", "IT strategy", "Risk register updates", "Architecture decisions"]
},
engage: {
title: "Engage",
desc: "Provides a good understanding of stakeholder needs, enables transparency, and maintains productive relationships. The primary contact point between the organization and its customers and users.",
examples: ["Service requests intake", "User feedback collection", "Customer communications", "SLA negotiation"]
},
design: {
title: "Design & Transition",
desc: "Ensures products and services continually meet stakeholder expectations for quality, costs, and time-to-market. Includes design, testing, and transitioning services into live operation.",
examples: ["Service design documents", "Test plans", "Release planning", "Training materials"]
},
build: {
title: "Obtain / Build",
desc: "Ensures service components are available when and where needed, meeting agreed specifications. Covers both acquiring from suppliers and building internally.",
examples: ["Software development", "Infrastructure provisioning", "Vendor procurement", "Configuration of components"]
},
deliver: {
title: "Deliver & Support",
desc: "Ensures services are delivered and supported according to agreed specifications and stakeholder expectations. Day-to-day operations, incident management, and service fulfillment.",
examples: ["Incident resolution", "Service desk operations", "Monitoring & alerting", "Fulfilling service requests"]
},
improve: {
title: "Improve",
desc: "Ensures continual improvement across all products, services, and practices. Works in parallel with all other activities — every component of the SVS should be subject to improvement at all times.",
examples: ["Post-incident reviews", "Performance trend analysis", "Process optimization", "Updating the CIR (Continual Improvement Register)"]
}
};
const panel = document.getElementById('svc-panel');
document.querySelectorAll('.itil4-svc-wrapper [data-svc]').forEach(el => {
el.addEventListener('click', function() {
const key = this.dataset.svc;
const d = info[key];
if (!d) return;
document.querySelectorAll('.itil4-svc-wrapper [data-svc]').forEach(e => e.classList.remove('active'));
this.classList.add('active');
const examples = d.examples.map(e => `<span>${e}</span>`).join('');
panel.innerHTML = `
<div class="panel-title">${d.title}</div>
<div class="panel-desc">${d.desc}</div>
<div class="panel-examples">${examples}</div>
`;
});
});
})();
</script>
</div>
<p><br><br></p>
<h2 id="the-4-dimensions">The 4 dimensions</h2>
<p>Every component of the SVS should be considered from 4 perspectives. Ignoring any of them is one of the most reliable ways to make a project fail.</p>
<table class="it">
<thead><tr><th>Dimension</th><th>What It Covers</th><th>Classic Failure Mode</th></tr></thead>
<tbody>
<tr><td><strong>Organizations and People</strong></td><td>Org structure, culture, roles, responsibilities</td><td>"We deployed the tool but nobody knows how to use it"</td></tr>
<tr><td><strong>Information and Technology</strong></td><td>Data, knowledge, tools, infrastructure</td><td>"The system works but we can't query the data we need"</td></tr>
<tr><td><strong>Partners and Suppliers</strong></td><td>External providers, contracts, integrations</td><td>"Our vendor went down and we had no fallback"</td></tr>
<tr><td><strong>Value Streams and Processes</strong></td><td>How work flows through the organization</td><td>"Everyone's busy but nothing ships"</td></tr>
</tbody>
</table>
<p>These 4 dimensions are interdependent. Migrating to a new monitoring tool (Information & Technology) means retraining people (Organizations & People), renegotiating vendor contracts (Partners & Suppliers), and redesigning on-call workflows (Value Streams & Processes).</p>
<p><br><br></p>
<h2 id="the-7-guiding-principles">The 7 guiding principles</h2>
<p>These are universal principles that guide decision-making in any circumstance. They apply regardless of which practices you're implementing.</p>
<p><strong>1. Focus on value</strong><br />
Every activity should link back to value for stakeholders. Before starting anything, ask: who benefits, and how? If you can't answer that, stop.</p>
<p><strong>2. Start where you are</strong><br />
Never ever assume you need to start from scratch. Assess what already works before designing something new. Most organizations already have something useful, the goal is to build on it, not erase it.</p>
<p><strong>3. Progress iteratively with feedback</strong><br />
Don't try to design the perfect solution upfront. Deliver in small increments, gather feedback, adjust. This is the same philosophy behind Agile sprints and DevOps Continuous Delivery.</p>
<p><strong>4. Collaborate and promote visibility</strong><br />
Work done in silos fails. Decisions made without the right people in the room fail. Make work visible: kanban boards, incident dashboards, change calendars; and involve stakeholders early, not at the end.</p>
<p><strong>5. Think and work holistically</strong><br />
Services are systems. A change to one component can have unexpected effects elsewhere. Consider end-to-end impact, not just the piece in front of you. <em>Lol!</em></p>
<p><strong>6. Keep it simple and practical</strong><br />
If a process step doesn't add value, eliminate it. If a tool creates more overhead than it saves, reconsider. Complexity is the enemy of reliability. Where I come from, we say <strong><em>keep it simple and stupid</em></strong>, see <a href="https://en.wikipedia.org/wiki/KISS_principle">here</a></p>
<p><strong>7. Optimize and automate</strong><br />
First, optimize the process (eliminate waste, reduce friction). Then, automate what remains. Automating a broken process gives you broken results, faster. The order matters.</p>
<p><br><br></p>
<h2 id="the-34-management-practices">The 34 management practices</h2>
<p>ITIL v4 defines <strong>34 practices</strong> designed for performing work or accomplishing an objective. They replace what ITIL v3 called "processes" (the shift reflects that a practice includes culture, tools, and knowledge, not just the steps).</p>
<p>They're organized into 3 categories:</p>
<table class="it">
<thead><tr><th>Category</th><th>Count</th><th>Examples</th></tr></thead>
<tbody>
<tr><td><strong>General management</strong></td><td>14</td><td>Risk management, knowledge management, project management, continual improvement, organizational change management</td></tr>
<tr><td><strong>Service management</strong></td><td>17</td><td>Incident management, problem management, change enablement, service desk, service level management</td></tr>
<tr><td><strong>Technical management</strong></td><td>3</td><td>Deployment management, infrastructure & platform management, software development & management</td></tr>
</tbody>
</table>
<p>The full list of <a href="https://itsm.tools/34-itil-4-management-practices/">34 ITIL 4 practices</a> exists for reference. No, you do not need to apply all 34 ITIL v4 practices. ITIL v4 is designed to be <strong>adopted and adapted</strong>. The core philosophy is to use <strong>only</strong> the practices that address your <strong>organization's specific needs, pain points, and goals.</strong></p>
<p>When you start, you need to identify your top 3 pain points (e.g., too many incidents, slow changes) to pick the right practices to implement first.</p>
<p>Here, we are almost starting from zero and we are already overwhelmed to the max. I would like to suggest 4 practices that, if we don't have them, we're operating blind:</p>
<ol start="0">
<li>IT asset management → What do we actually own?</li>
<li>Service request management → What are our users asking for?</li>
<li>Incident Management → What is broken right now?</li>
<li>Change Enablement → What is changing and why?</li>
</ol>
<p>The 4 together cover the 4 basic operational questions any IT team needs answered at any moment. Everything else can be phased in.</p>
<p><br><br></p>
<h2 id="incident-management">Incident management</h2>
<p><strong>Incident management</strong> is the practice responsible for managing the lifecycle of all incidents: unplanned interruptions or reductions in the quality of a service. Before we go further, let's nail the vocabulary:</p>
<table class="it">
<thead><tr><th>Term</th><th>Definition</th></tr></thead>
<tbody>
<tr><td><strong>Error</strong></td><td>A flaw in a component that could cause incidents</td></tr>
<tr><td><strong>Incident</strong></td><td>An unplanned interruption or reduction in service quality</td></tr>
<tr><td><strong>Problem</strong></td><td>The cause (known or unknown) of one or more incidents</td></tr>
<tr><td><strong>Known error</strong></td><td>A problem that has been successfully analyzed, and a workaround or permanent solution has been identified</td></tr>
<tr><td><strong>Root Cause</strong></td><td>The fundamental reason behind a problem</td></tr>
</tbody>
</table>
<p>These are distinct. An <strong>incident</strong> is what you respond to at 2am. A <strong>problem</strong> is what you investigate to prevent the next one. Conflating them is how teams fix the same incident repeatedly without ever addressing what's actually causing it.</p>
<h3 id="the-7-step-incident-lifecycle">The 7-Step incident lifecycle</h3>
<table class="it">
<thead><tr><th>Step</th><th>Action</th></tr></thead>
<tbody>
<tr><td>1</td><td><strong>Identification</strong>: monitoring alert, user report, or support ticket</td></tr>
<tr><td>2</td><td><strong>Logging</strong>: full record: timestamp, description, affected service, reporter</td></tr>
<tr><td>3</td><td><strong>Categorization</strong>: what type of incident? which service/component?</td></tr>
<tr><td>4</td><td><strong>Prioritization</strong>: how urgent? how impactful?</td></tr>
<tr><td>5</td><td><strong>Diagnosis</strong>: initial investigation, probable cause</td></tr>
<tr><td>6</td><td><strong>Resolution</strong>: fix applied, service restored</td></tr>
<tr><td>7</td><td><strong>Closure</strong>: confirm resolution, document lessons, update knowledge base</td></tr>
</tbody>
</table>
<h3 id="incident-priority-matrix">Incident Priority Matrix</h3>
<p>Priority is determined by <strong>Impact</strong> (how many users/processes are affected) and <strong>Urgency</strong> (how quickly the business needs resolution). The matrix:</p>
<table class="it">
<thead><tr><th></th><th>High Urgency</th><th>Medium Urgency</th><th>Low Urgency</th></tr></thead>
<tbody>
<tr><td><strong>High Impact</strong></td><td><strong>P1</strong></td><td>P2</td><td>P3</td></tr>
<tr><td><strong>Medium Impact</strong></td><td>P2</td><td>P3</td><td>P3</td></tr>
<tr><td><strong>Low Impact</strong></td><td>P3</td><td>P3</td><td>P4</td></tr>
<tr><td><strong>Trivial Impact</strong></td><td>P4</td><td>P4</td><td>P4</td></tr>
</tbody>
</table>
<p>Priority codes (not to be confused with Biosafety Levels):</p>
<ul>
<li><strong>P1 (Critical)</strong>: Production down, major business impact. Immediate response, all hands.</li>
<li><strong>P2 (High)</strong>: Significant service degradation. Fast response, escalation ready.</li>
<li><strong>P3 (Medium)</strong>: Partial impact, workaround available. Normal SLA response.</li>
<li><strong>P4 (Low)</strong>: Minimal impact, cosmetic or edge case. Can be scheduled.</li>
</ul>
<p><strong>NB: BSL (Biosafety Level) is the standard, modern terminology, while P (P1, P2, P3, P4) is an older designation standing for Pathogen/Protection level.</strong></p>
<h3 id="escalation">Escalation</h3>
<p>The process within incident and service request management used to transfer issues to higher levels of authority or expertise when they cannot be resolved within normal, front-line support procedures.</p>
<p>When should we escalate ? When the incident exceed SLA timeframes, require higher expertise, or need more management authority.</p>
<p>There are 2 types:</p>
<ul>
<li><strong>Functional escalation</strong>: pass to someone with more technical expertise</li>
<li><strong>Hierarchical escalation</strong>: involve management when impact warrants it or when SLA breach is imminent</li>
</ul>
<p>A <strong>crisis</strong> is when an incident has business-wide consequences that require executive involvement. Knowing when a P1 becomes a crisis (and who to call) should be documented <em>before</em> the crisis, not during it.</p>
<p><br><br></p>
<h2 id="change-enablement">Change enablement</h2>
<p><strong>Change Enablement</strong> manages the lifecycle of all changes to maximize the probability of success and minimize disruption.</p>
<p>ITIL v4 recognizes 3 change types:</p>
<table class="it">
<thead><tr><th>Type</th><th>Risk</th><th>Authorization</th><th>Examples</th></tr></thead>
<tbody>
<tr><td><strong>Standard</strong></td><td>Low, well-understood</td><td>Pre-authorized</td><td>Antivirus update, user account creation</td></tr>
<tr><td><strong>Normal</strong></td><td>Variable</td><td>Change authority / CAB review</td><td>Application deployment, infrastructure change</td></tr>
<tr><td><strong>Emergency</strong></td><td>High urgency</td><td>Emergency authority (fast-track)</td><td>Critical security patch, production hotfix</td></tr>
</tbody>
</table>
<h3 id="the-cab-change-advisory-board">The CAB (Change Advisory Board)</h3>
<p>The <strong>CAB</strong> is a group that reviews and authorizes significant normal changes. Typical members: IT operations, security, business stakeholders, and the change owner. The CAB doesn't exist to slow things down, it exists to catch problems before they become incidents. It is the organizational mechanism that <em>prevents the finance team from being unable to run payroll on a Friday.</em></p>
<p>A proper CAB dossier for a normal change includes:</p>
<ul>
<li><strong>Impact</strong>: which services, users, and systems are affected?</li>
<li><strong>Risk</strong>: what's the probability of failure, and what's the impact if it fails?</li>
<li><strong>Rollback plan</strong>: how do we revert cleanly if the change fails? (Tested rollback, not theoretical)</li>
<li><strong>Post-implementation review</strong>: how do we verify success after deployment?</li>
</ul>
<p><br><br></p>
<h2 id="continual-improvement">Continual improvement</h2>
<p>Continual improvement is both a guiding principle and a dedicated practice in ITIL 4. It ensures the organization continuously improves its services, practices, and the SVS itself.</p>
<p>The <strong>Continual Improvement Register (CIR)</strong> is the operational tool: a log of all improvement ideas, their status, priority, and outcomes. Think of it as a backlog specifically for operational improvements, not features, not incidents, but <em>how we work</em>.</p>
<h3 id="the-7-step-improvement-model">The 7-Step improvement model</h3>
<p>This model is cyclical by design. Reaching step 7 doesn't mean you're done, it means you start over with a new vision.</p>
<table class="it">
<thead><tr><th>Step</th><th>Question to Answer</th></tr></thead>
<tbody>
<tr><td>1</td><td><strong>What is the vision?</strong> Align the improvement to business strategy</td></tr>
<tr><td>2</td><td><strong>Where are we now?</strong> Baseline assessment: metrics, maturity, current state</td></tr>
<tr><td>3</td><td><strong>Where do we want to be?</strong> Define target state and measurable success criteria</td></tr>
<tr><td>4</td><td><strong>How do we get there?</strong> Build the improvement plan</td></tr>
<tr><td>5</td><td><strong>Take action.</strong> Execute iteratively</td></tr>
<tr><td>6</td><td><strong>Did we get there?</strong> Measure against success criteria</td></tr>
<tr><td>7</td><td><strong>How do we keep the momentum?</strong> Embed the change, celebrate wins, start the cycle again</td></tr>
</tbody>
</table>
<h3 id="measuring-improvement-sla-vs-xla-and-beyond">Measuring improvement: SLA vs XLA and beyond</h3>
<table class="it">
<thead><tr><th>Metric</th><th>What It Measures</th></tr></thead>
<tbody>
<tr><td><strong>KPI</strong> (Key Performance Indicator)</td><td>Technical performance against defined targets</td></tr>
<tr><td><strong>SLA</strong> (Service Level Agreement)</td><td>Contractual service commitments (uptime, response times)</td></tr>
<tr><td><strong>OKR</strong> (Objective and Key Result)</td><td>Strategic goal achievement across a period</td></tr>
<tr><td><strong>XLA</strong> (Experience Level Agreement)</td><td>User satisfaction and actual experience quality</td></tr>
<tr><td><strong>CIR</strong></td><td>Improvement pipeline health and throughput</td></tr>
</tbody>
</table>
<p><strong>SLA vs XLA</strong> deserves its own paragraph. An SLA measures <em>what the system does</em> (was uptime ≥ 99.9%?). An <a href="https://itsm.tools/what-exactly-are-xlas-and-how-do-you-use-them/">XLA</a> measures <em>how the user felt about it</em>. You can meet every SLA and still have furious users, because the system was slow even within the threshold, or the error messages were unhelpful, or the support experience was frustrating. XLAs align IT metrics with actual business outcomes, not just technical compliance.</p>
<h3 id="3-axes-of-improvement">3 Axes of improvement</h3>
<ul>
<li><strong>Strategic</strong>: organization-wide, long-term (e.g., <em>"reach 95% automated change deployment by end of year"</em>)</li>
<li><strong>Tactical</strong>: team/process level, medium-term (e.g., <em>"reduce MTTR on P1 incidents by 40% this quarter"</em>)</li>
<li><strong>Operational</strong>: day-to-day (e.g., <em>"update the incident categorization taxonomy this sprint"</em>)</li>
</ul>
<p><br><br></p>
<h2 id="governance-and-risk-management">Governance and Risk Management</h2>
<p><strong>Governance</strong> sits at the top of the SVS. It provides direction and ensures the organization stays aligned with its objectives and obligations. Three functions:</p>
<ul>
<li><strong>Direct</strong>: Set strategy, policies, and organizational objectives</li>
<li><strong>Evaluate</strong>: Assess organizational performance, risks, and compliance</li>
<li><strong>Monitor</strong>: Track adherence to policies and verify outcomes</li>
</ul>
<p>For service management, governance ensures that the SVS is accountable: decisions are traceable, risks are understood, and exceptions are handled systematically.</p>
<p><strong>Risk Management</strong> in ITIL follows a straightforward formula:</p>
<blockquote>
<p><strong>Risk = Probability × Impact</strong></p>
</blockquote>
<p>Three responses:</p>
<ol>
<li><strong>Mitigate</strong>: reduce probability or impact (implement redundancy, add monitoring, require peer review)</li>
<li><strong>Transfer</strong>: shift the risk to another party (insurance, contractual SLAs with vendors, outsourcing)</li>
<li><strong>Accept</strong>: acknowledge the risk and decide to live with it (for low probability, low impact scenarios)</li>
</ol>
<p>Risk isn't inherently bad. The problem is <strong>unacknowledged risk</strong>, for example: changes deployed without a risk assessment, dependencies unknown until they fail, vulnerabilities known but not documented. Risk management doesn't eliminate uncertainty; it makes it visible and traceable.</p>
<p><br><br></p>
<h2 id="itil-4-agile-devops-lean">ITIL 4 ↔ Agile, DevOps, Lean</h2>
<p>One of the major improvements in ITIL v4 over its predecessors is the explicit integration with modern practices. The framework doesn't compete with DevOps or Agile. It provides the governance layer that makes them sustainable at scale.</p>
<table class="it">
<thead><tr><th>ITIL 4 Concept</th><th>Agile Equivalent</th><th>DevOps Equivalent</th><th>Lean Equivalent</th></tr></thead>
<tbody>
<tr><td>Service Value System</td><td>Product Delivery Flow</td><td>CI/CD Pipeline</td><td>Value Stream</td></tr>
<tr><td>Continual Improvement</td><td>Sprint Retrospective</td><td>Blameless Post-mortem</td><td>Kaizen</td></tr>
<tr><td>Change Enablement</td><td>Sprint Planning</td><td>Deployment Pipeline</td><td>Flow Management</td></tr>
<tr><td>Incident Management</td><td>Bug Sprint</td><td>Incident Response</td><td>A3 Problem Solving</td></tr>
<tr><td>Service Request</td><td>Backlog Item</td><td>Work Item</td><td>Pull System</td></tr>
<tr><td>SLA / XLA</td><td>Definition of Done</td><td>SLO / SLI</td><td>Quality Standards</td></tr>
<tr><td>CAB Review</td><td>Sprint Review</td><td>Deployment Gate</td><td>Go/No-Go Decision</td></tr>
<tr><td>Problem Management</td><td>Root Cause Analysis</td><td>Post-mortem</td><td>5 Whys</td></tr>
<tr><td>Service Desk</td><td>Product Owner (ops)</td><td>L1/L2 Support</td><td>Andon Cord</td></tr>
</tbody>
</table>
<p>The <a href="https://www.axelos.com/resource-hub">official ITIL 4 and DevOps resources</a> make the integration case explicitly. The short version: <strong>ITIL handles governance and accountability; DevOps handles speed and flow. You need both.</strong> Speed without governance creates chaos. Governance without speed creates bureaucracy. ITIL v4 was redesigned precisely to not be the bottleneck.</p>
<p><br><br></p>
<h2 id="case-study">Case study</h2>
<p>Here is what ITIL v4 looks like applied to a real IT direction inside a biomedical research center in Senegal. Any resemblance to real persons, living or dead, is <em>purely coincidental.</em></p>
<h3 id="part-1-service-catalog">Part 1 — Service catalog</h3>
<p>The official list of what the IT direction provides, to whom, under what terms. It is the operational contract between the IT team and the rest of the organization. Without it, users don't know what to ask for, IT doesn't know what they're accountable for, and incidents pile up in a shared mailbox nobody consistently monitors.</p>
<p>Below: 5 services covering the majority of what a research institute IT direction actually handles day-to-day.</p>
<table class="it">
<thead>
<tr>
<th>Service</th>
<th>Description</th>
<th>Channel</th>
<th>SLA — Response / Resolution</th>
<th>Responsible</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>User account provisioning</strong></td>
<td>Creation, modification, or deactivation of accounts (Active Directory, email, VPN, application access). Covers onboarding and offboarding.</td>
<td>ITSM ticketing portal / HR workflow trigger</td>
<td>4h / 1 business day</td>
<td>Sysadmin — Identity & Access team</td>
</tr>
<tr>
<td><strong>Workstation support</strong></td>
<td>Hardware diagnosis and repair, OS reinstall, peripheral setup (printers, scanners, external drives). On-site or remote via support session.</td>
<td>Ticketing portal — phone for P1/P2 only</td>
<td>P3: 4h / 2 days — P4: next business day / 5 days</td>
<td>IT Support technician</td>
</tr>
<tr>
<td><strong>Application support</strong></td>
<td>Functional support for lab and corporate applications: LIMS, ERP (finance, HR), syndromic surveillance platform, email (M365). Includes access requests and usage questions.</td>
<td>Ticketing portal, categorized by application</td>
<td>P2: 2h / 4h — P3: 4h / 1 day — P4: 1 day / 3 days</td>
<td>Application owner + IT Operations</td>
</tr>
<tr>
<td><strong>VPN & remote access</strong></td>
<td>Setup, configuration, and troubleshooting of remote access (VPN client, MFA token). Covers new setups for approved staff and incident resolution for existing connections.</td>
<td>Ticketing portal — escalation via phone if blocking remote work</td>
<td>New setup: 1 day / 2 days — Incident: 2h / 4h</td>
<td>Network team</td>
</tr>
<tr>
<td><strong>Backup & data recovery</strong></td>
<td>Scheduled backups of research data, institutional databases, and user file shares. Restore requests for accidental deletions or corruption. Covers file-level and full-system restores.</td>
<td>Ticketing portal — critical restores via phone (P1/P2)</td>
<td>P1 restore: 1h acknowledged / 4h RTO — P3 file restore: 4h / 1 day</td>
<td>Sysadmin — Infrastructure team</td>
</tr>
</tbody>
</table>
<hr />
<h3 id="part-2-rfc-cab-dossier-lims-major-version-upgrade">Part 2 — RFC & CAB Dossier: LIMS major version upgrade</h3>
<p>A <strong>Request for Change (RFC)</strong> is the formal input to the Change Enablement practice. For a Normal or a Major change, it triggers a full CAB review before any action is taken. Here is what a complete dossier looks like.</p>
<p><strong>Context</strong>: Institut Patrick de Kinshasa uses a LIMS (Laboratory Information Management System) to track samples, tests, and results across all its labs. The vendor has released version 4.0, which drops support for the current version 3.x at end of year. The upgrade is not optional, but it touches sample data, active test runs, and integrations with the syndromic surveillance platform.</p>
<hr />
<h4 id="rfc-identification">RFC identification</h4>
<table class="it">
<thead><tr><th>Field</th><th>Value</th></tr></thead>
<tbody>
<tr><td><strong>RFC Number</strong></td><td>RFC-2026-014</td></tr>
<tr><td><strong>Title</strong></td><td>LIMS major version upgrade: v3.8 → v4.0</td></tr>
<tr><td><strong>Change type</strong></td><td>Normal — Major</td></tr>
<tr><td><strong>Requestor</strong></td><td>Head of IT Operations</td></tr>
<tr><td><strong>Application owner</strong></td><td>Director of Laboratory Services</td></tr>
<tr><td><strong>Submission date</strong></td><td>2026-05-03</td></tr>
<tr><td><strong>Requested deployment window</strong></td><td>2026-05-23, Saturday 22:00 → Sunday 06:00 (WAT)</td></tr>
<tr><td><strong>CAB review date</strong></td><td>2026-05-13</td></tr>
</tbody>
</table>
<h4 id="justification">Justification</h4>
<p>The current LIMS v3.x reaches vendor end-of-life on 2026-12-31. After that date: no security patches, no bug fixes, no vendor support. Version 4.0 also introduces a native REST API required by the next release of the syndromic surveillance platform. This change is driven by compliance (security posture) and technical dependency (surveillance platform roadmap) simultaneously.</p>
<hr />
<h4 id="impact-assessment">Impact assessment</h4>
<table class="it">
<thead><tr><th>Dimension</th><th>Details</th></tr></thead>
<tbody>
<tr><td><strong>Systems affected</strong></td><td>LIMS application servers (lims-prod-01, lims-prod-02), LIMS PostgreSQL database (lims-db-01), integration bridge with surveillance platform</td></tr>
<tr><td><strong>Users affected</strong></td><td>~65 lab technicians and researchers across 4 departments: Virology, Bacteriology, Epidemiology, Quality</td></tr>
<tr><td><strong>Business processes affected</strong></td><td>Sample reception, test assignment, result entry, result validation, report generation, QC workflows</td></tr>
<tr><td><strong>Downstream services</strong></td><td>Surveillance platform reads LIMS results via API — will operate in read-only degraded mode during the window</td></tr>
<tr><td><strong>Duration of impact</strong></td><td>LIMS unavailable ~8 hours; all impact contained within the Saturday night window</td></tr>
<tr><td><strong>Data at risk</strong></td><td>~120,000 sample records, 3 years of test history. No patient PII in LIMS (samples referenced by anonymous ID).</td></tr>
</tbody>
</table>
<hr />
<h4 id="risk-assessment">Risk assessment</h4>
<table class="it">
<thead><tr><th>Risk</th><th>Probability</th><th>Impact</th><th>Score</th><th>Mitigation</th></tr></thead>
<tbody>
<tr><td>DB migration script fails mid-run, leaving schema in inconsistent state</td><td>Low</td><td>Critical</td><td>🔴 High</td><td>Full DB snapshot before migration; script tested 3× on staging with a copy of production data</td></tr>
<tr><td>v4.0 breaks custom report templates (Quality dept uses non-standard format)</td><td>Medium</td><td>Medium</td><td>🟡 Medium</td><td>All 12 templates validated on staging; Quality dept sign-off obtained before CAB submission</td></tr>
<tr><td>Surveillance platform integration fails after API change</td><td>Low</td><td>High</td><td>🟡 Medium</td><td>Integration tested on staging against v4.0 API; surveillance team on standby Sunday 00:30</td></tr>
<tr><td>Deployment overruns window, runs into Monday lab opening at 07:30</td><td>Low</td><td>High</td><td>🟡 Medium</td><td>Hard rollback decision point at 04:00 — if not complete by then, rollback regardless of progress</td></tr>
<tr><td>UI changes slow down lab technicians Monday morning</td><td>Medium</td><td>Low</td><td>🟢 Low</td><td>30-min walkthrough session Monday 08:00; quick reference card distributed Friday</td></tr>
</tbody>
</table>
<hr />
<h4 id="implementation-plan">Implementation plan</h4>
<table class="it">
<thead><tr><th>Time (WAT)</th><th>Step</th><th>Responsible</th><th>Checkpoint</th></tr></thead>
<tbody>
<tr><td>Sat 21:30</td><td>Pre-deployment checklist: verify backup completed, confirm no active test runs, notify on-call lab manager</td><td>IT Ops lead</td><td>✓ required to proceed</td></tr>
<tr><td>Sat 22:00</td><td>Put LIMS in maintenance mode (users see scheduled maintenance banner)</td><td>Sysadmin</td><td></td></tr>
<tr><td>Sat 22:05</td><td>Full PostgreSQL dump to backup server + verify checksum</td><td>DBA</td><td>✓ required to proceed</td></tr>
<tr><td>Sat 22:30</td><td>Stop LIMS application services on both nodes</td><td>Sysadmin</td><td></td></tr>
<tr><td>Sat 22:35</td><td>Run database schema migration script (estimated: 45 min)</td><td>DBA</td><td>✓ required to proceed</td></tr>
<tr><td>Sat 23:20</td><td>Deploy v4.0 application package on lims-prod-01 (primary node)</td><td>Sysadmin</td><td></td></tr>
<tr><td>Sat 23:40</td><td>Run automated smoke test suite against lims-prod-01</td><td>IT Ops lead</td><td>✓ required to proceed</td></tr>
<tr><td>Sun 00:00</td><td>Deploy v4.0 on lims-prod-02, verify cluster synchronization</td><td>Sysadmin</td><td></td></tr>
<tr><td>Sun 00:30</td><td>Test surveillance platform integration against new v4.0 API</td><td>Surveillance team</td><td>✓ required to proceed</td></tr>
<tr><td>Sun 01:00</td><td>Manual validation by on-call lab manager: create test sample, enter result, generate report</td><td>Lab manager</td><td>✓ required to proceed</td></tr>
<tr><td>Sun 01:30</td><td>Remove maintenance mode, monitor application logs for 30 min</td><td>IT Ops lead</td><td></td></tr>
<tr><td>Sun 02:00</td><td>Deployment declared successful — confirmation sent to all stakeholders</td><td>IT Ops lead</td><td></td></tr>
</tbody>
</table>
<p><strong>Hard rollback decision point: Sunday 04:00.</strong> If not complete and validated by then, rollback is initiated regardless of progress. No exceptions — the lab opens at 07:30.</p>
<hr />
<h4 id="rollback-procedure">Rollback procedure</h4>
<p>Triggered if: any required checkpoint fails and cannot be resolved within 30 minutes, OR the 04:00 hard deadline is reached.</p>
<ol>
<li>Stop all LIMS application services on both nodes</li>
<li>Drop the migrated database: <code>dropdb lims_production</code></li>
<li>Restore from the pre-migration dump: <code>pg_restore -d lims_production /backup/lims-premig-20260523.dump</code></li>
<li>Verify integrity: compare row counts for <code>samples</code>, <code>tests</code>, <code>results</code> tables against the pre-migration snapshot</li>
<li>Redeploy the v3.8 package from artifact registry on both nodes</li>
<li>Restart application services, verify cluster health</li>
<li>Remove maintenance mode, run the same smoke test suite</li>
<li>Notify all stakeholders: rollback completed, root cause analysis scheduled Monday</li>
</ol>
<p>Estimated rollback duration: <strong>45 minutes.</strong> The 04:00 decision point leaves 2.5 hours of margin before lab opening.</p>
<hr />
<h4 id="communication-plan">Communication plan</h4>
<table class="it">
<thead><tr><th>Audience</th><th>Message</th><th>When</th><th>Channel</th></tr></thead>
<tbody>
<tr><td>All LIMS users</td><td>Scheduled maintenance — LIMS unavailable Saturday 22:00 → Sunday 06:00</td><td>Friday 2026-05-22, 14:00</td><td>Email + in-app banner</td></tr>
<tr><td>Lab managers</td><td>Upgrade rationale, v4.0 UI changes, quick reference card attached</td><td>Friday 2026-05-22, 14:00</td><td>Email</td></tr>
<tr><td>Surveillance team</td><td>API changes summary, integration test results, standby request for Sunday 00:30</td><td>Thursday 2026-05-21</td><td>Direct message + ticket</td></tr>
<tr><td>All stakeholders</td><td>Outcome (success or rollback) + next steps</td><td>Sunday 2026-05-24, by 06:00</td><td>Email</td></tr>
</tbody>
</table>
<hr />
<h4 id="post-implementation-review">Post-implementation review</h4>
<p>Scheduled: <strong>Monday 2026-05-25, 10:00</strong> / 30 minutes. Attendees: IT Ops lead, application owner, one representative per affected lab department.</p>
<p>Agenda:</p>
<ol>
<li>Did deployment complete within window? If not: what caused the overrun?</li>
<li>Were all required checkpoints met? Document any deviations or waivers.</li>
<li>User-reported issues since go-live (collected Monday morning, before the review)</li>
<li>Was rollback triggered? If yes: full root cause analysis before next major change</li>
<li>Were Application Support SLAs met during the post-go-live monitoring period?</li>
<li>Update the LIMS runbook with any new procedures discovered during the upgrade</li>
<li>Close RFC-2026-014 in the ITSM tool; status: <em>Successful</em> or <em>Successful with deviations</em></li>
</ol>
<p>The output feeds directly into the <strong>Continual Improvement Register</strong>: anything harder than expected, any gap in the rollback procedure, any user adoption friction; logged as improvement items before the next major change.</p>
<p><br><br></p>
<h2 id="tooling">Tooling</h2>
<p>No tool enforces ITIL by itself. The process has to exist first, the tool just reduces friction. That said, the right tooling makes the difference between a practice that lives on paper and one that teams actually follow.</p>
<p>A word on scope: <strong>ServiceNow</strong> is the de facto enterprise standard and covers nearly everything in the table below. It is also expensive, slow to configure, and overkill for most organizations below a few hundred IT staff.</p>
<p>The open source column below deserves serious consideration. <strong>GLPI</strong> in particular covers incident, change, CMDB, and knowledge management in a single self-hosted package.</p>
<table class="it">
<thead>
<tr>
<th></th>
<th>ITSM Platform</th>
<th>Incident Detection & Alerting</th>
<th>Incident Response</th>
<th>Knowledge Management</th>
<th>Change Tracking</th>
<th>Continual Improvement</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Suggested tools</strong></td>
<td>ServiceNow, Jira Service Management, Freshservice</td>
<td>PagerDuty, OpsGenie, incident.io</td>
<td>FireHydrant, Rootly</td>
<td>Confluence, Notion, GitBook</td>
<td>Jira, LinearB</td>
<td>Sleuth, Datadog, LinearB</td>
</tr>
<tr>
<td><strong>Free & open source</strong></td>
<td><a href="https://www.glpi-project.org/">GLPI</a>, <a href="https://www.combodo.com/itop">iTop</a>, <a href="https://www.znuny.org/en">Znuny</a></td>
<td><a href="https://prometheus.io/">Prometheus + Alertmanager</a>, <a href="https://www.zabbix.com/">Zabbix</a></td>
<td>—</td>
<td><a href="https://js.wiki/">Wiki.js</a>, <a href="https://www.getoutline.com/">Outline</a>, <a href="https://www.bookstackapp.com/">BookStack</a></td>
<td><a href="https://plane.so/">Plane</a>, <a href="https://about.gitea.com/">Gitea</a> / <a href="https://forgejo.org/">Forgejo</a></td>
<td><a href="https://github.com/dora-team/fourkeys">Four Keys (Google)</a></td>
</tr>
</tbody>
</table>
<p><br><br></p>
<h2 id="itil-v5">ITIL v5</h2>
<p>We've been taught during the training ITIL v5 was launched in february 2026. I tried to watch this <a href="https://www.youtube.com/watch?v=8IxxdEgE_OE">video</a> to have an overview of what is ITIL v5. The only thing I understood was: "The new pace of changes today driven by <a href="#">cloud</a>, <a href="#">ai</a>, digital products, experience-focused services, demand a new way of thinking." ITIL v5 answers that need. <em>Phew!</em></p>
<p><br><br></p>
<h2 id="conclusion">Conclusion</h2>
<p>ITIL v4 is a framework providing a <strong>vocabulary</strong> and a <strong>structure for conversations</strong> your organization was probably already having informally: <em>Who decides what gets changed? How do we prioritize when everything is on fire? How do we know we're actually improving?</em></p>
<p>The most valuable takeaways from this training:</p>
<ol>
<li><strong>Value is co-created, not delivered.</strong> You can deploy perfect infrastructure and still fail if users can't get value from it.</li>
<li><strong>SLA ≠ user happiness</strong>. XLAs measure what actually matters to the business.</li>
<li><strong>ITIL v4 and DevOps are complementary.</strong> ITIL provides the accountability layer that makes continuous delivery safe at organizational scale.</li>
<li><strong>Adopt and adapt.</strong> Apply the practices that solve real problems, and leave the rest on the shelf until you need them.</li>
</ol>
<p>If your systems go down and nobody knows who to call, what changed yesterday, or how long things have been broken, that's not a technical problem. That's a service management problem. ITIL v4 gives you the vocabulary to fix it.</p>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>Congratulations, you made it to the end. If you enjoyed this, please <em>like, share & subscriiiibe</em>. You might also like my amazing post about <a href="https://nskm.xyz/posts/aws-1/">AWS fundamentals</a> or <a href="https://nskm.xyz/posts/syndromic-surveillance/">how I'm helping save lives in West Africa using free software</a>.</p>
<p>This article is a summary of a three-day training course. Impossible to explore all the aspects of ITIL v4 in such a short post. Here are some useful links to lean more:</p>
<ul>
<li><a href="https://www.peoplecert.org/browse-certifications/it-governance-and-service-management/ITIL-1">ITIL 4 official certification path</a></li>
<li><a href="https://itsm.tools/itil-guiding-principles/">ITIL 4 Guiding Principles explained</a></li>
<li><a href="https://itsm.tools/34-itil-4-management-practices/">34 ITIL 4 management practices</a></li>
<li><a href="https://itsm.tools/what-exactly-are-xlas-and-how-do-you-use-them/">What are XLAs and how do you use them?</a></li>
<li><a href="https://itrevolution.com/product/the-phoenix-project/">The Phoenix project</a></li>
<li><a href="https://sc.edu/about/offices_and_divisions/ehs/research_and_laboratory_safety/biological_safety/biosafety_level_criteria/">Biosafety Levels</a></li>
</ul>
<p>Even more:</p>
<ul>
<li><a href="https://www.gamingworks.nl/business-simulations/apollo-13/">Apollo 13 – An ITSM Case Experience (GamingWorks)</a></li>
<li><a href="https://www.gamechanginginsight.com/itil-in-play">GameChangingInsight ITIL simulations</a></li>
<li><a href="https://d12.github.io/itil-quiz/">ITIL v4 quiz (open source)</a></li>
<li><a href="https://purplegriffon.com/quizzes/itil-4-foundation-quiz-questions-and-answers">Purple Griffon ITIL 4 quiz</a></li>
<li><a href="https://www.careersmarter.com/itil-4-foundation-exam-questions/">Career Smarter mock exam</a></li>
<li><a href="https://www.processexam.com/peoplecert/peoplecert-itil-4-foundation-certification-exam-sample-questions">PeopleCert official sample questions</a></li>
</ul>
From rsync to CI/CD: automating a static site deployment2026-03-05T00:00:00+00:002026-03-05T00:00:00+00:00
Unknown
https://nskm.xyz/posts/mvln2/<img style="display: block; margin: 0 auto; width: 700px" src="/images/praying_ci_god.jpeg" alt="Praying the CI/CD god" title="Praying the CI/CD god"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers</h2>
<ol start="0">
<li>
<p><strong>Opinions</strong>: The views expressed here are my own unless otherwise specified. They do not represent any employer, organization, or affiliated group.</p>
</li>
<li>
<p><strong>Scope</strong>: This case study documents my personal journey automating a static Zola site. Your infrastructure needs may differ.</p>
</li>
<li>
<p><strong>Audience</strong>: This article assumes DevOps familiarity. You should be comfortable with Docker, shell scripts, git workflows, and Linux administration.</p>
</li>
<li>
<p><strong>Testing environment</strong>: Everything in this article was tested on Ubuntu (WSL2 on Windows) running locally, and deployed to the cheapest DigitalOcean.</p>
</li>
<li>
<p>Thanks a million to <a href="https://morgan.zoemp.be/about/">Morgan</a> alias <em>SansGuidon</em> online. He took the time to tell me that I broke my atom.xml file during this experiment.</p>
</li>
</ol>
<br>
<br>
<h2 id="hook">Hook</h2>
<p>If I recall correctly, I started writing tech articles somewhere around 2014. During all those years, my deployment workflow was delightfully simple: some Markdown files, two <code>make</code> commands, and my static site was live. It worked. It was fast. It required zero external infrastructure.</p>
<p>The irony? I spent years and years toying with CI/CD pipelines for employers, using all the modern <em>(and all the very old)</em> tooling. Yet my <em>own</em> site ran like it was 2005. <em>What a world!</em></p>
<p>I sincerely believe that there was no good reason to change. I only did it for fun and to entertain myself a little (<em>and maybe, maybe, to improve things a little bit</em>). This is the story of how I did it: from a <code>git push</code> on my local machine to a fully built, containerized, deployed site on a fresh server, with automatic TLS. <a href="https://www.youtube.com/watch?v=82-P9ZNYnik">Music \o/</a></p>
<hr />
<h2 id="toc">ToC</h2>
<ol>
<li><a href="https://nskm.xyz/posts/mvln2/#journey">The journey: where we were vs where we are going</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#gitlab-ci">GitLab CI pipeline</a>
<ul>
<li><a href="https://nskm.xyz/posts/mvln2/#gitlab-ci-config"> The GitLab CI configuration file</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#gitlab-ci-vars"> Required GitLab CI/CD variables</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#gitlab-registry"> GitLab container registry</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#notification"> Notification mechanism: GitLab CI SSHes directly into VPS</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/mvln2/#container-image">The Container image</a>
<ul>
<li><a href="https://nskm.xyz/posts/mvln2/#dockerfile"> The multi-stage Container file</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#nginx-config"> The Nginx configuration file</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/mvln2/#vps-provisioning">The VPS provisioning</a>
<ul>
<li><a href="https://nskm.xyz/posts/mvln2/#droplet"> Droplet recommendation</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#provisioning-steps"> Provisioning steps</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#vps-hardening"> VPS security hardening</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#doctl"> Going further: one-command provisioning with doctl</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#deploy-script"> Deploy script</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#podman-critical"> Critical: rootless Podman and SSH session lifecycle</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/mvln2/#ssl-tls">SSL & Let's Encrypt</a>
<ul>
<li><a href="https://nskm.xyz/posts/mvln2/#caddy"> Decision: Caddy</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#letsencrypt-ip"> Let's Encrypt IP address certificates</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/mvln2/#operations">Operations</a>
<ul>
<li><a href="https://nskm.xyz/posts/mvln2/#rollback"> Rollback procedure</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#image-scanning"> Container image security scanning</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#monitoring"> Uptime monitoring</a></li>
<li><a href="https://nskm.xyz/posts/mvln2/#zero-downtime"> Zero-downtime deploys</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/mvln2/#more">More on this topic</a></li>
</ol>
<p><br><br></p>
<h2 id="journey">The journey</h2>
<h3 id="where-i-was-simple-short-deployment">Where I was, simple/short deployment</h3>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>publish:
</span><span> zola build
</span><span>
</span><span>rsync_deploy_site:
</span><span> zola build
</span><span> find public -name "*.xml" -o -name "*.html" | xargs sed -i -e '/\&lt;!--.*--\&gt;/d'
</span><span> rsync -P -a -v --recursive -e "ssh -p $(SSH_PORT)" $(OUTPUTDIR)/ $(SSH_NICK):$(SSH_SITE_DIR) --cvs-exclude
</span><span>
</span><span>.PHONY: help publish rsync_deploy_site
</span></code></pre>
<p>Simple deployment because it involved 3 tools. Short deployment because there was a straight, direct path between my laptop and the server. I built the site locally and pushed the output directly to the VPS via rsync over SSH. The Makefile orchestrates everything — it runs <code>zola build</code> and then calls <code>rsync</code> to copy the <code>public/</code> directory to the server. That's it.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>[Local Machine]
</span><span> |
</span><span> | 1. make publish
</span><span> | └─ zola build → generates public/
</span><span> |
</span><span> | 2. make rsync_deploy_site
</span><span> | └─ rsync -a public/ mlvn@1.2.3.4:/home/mlvn/public (via SSH, port 1022)
</span><span> |
</span><span> v
</span><span>[DigitalOcean VPS]
</span><span> /home/mlvn/public/ ← raw static files on disk that are served by Nginx
</span><span> |
</span><span> v
</span><span>[Nginx/web server on VPS]
</span><span> Handles TLS (Let's Encrypt, auto-renewed)
</span><span> serves files directly from disk
</span><span> |
</span><span> v
</span><span>[Visitor's browser]
</span><span> https://nskm.xyz
</span></code></pre>
<p>There were no external dependencies, no container images to build, no container registries to manage, no GitLab CI runners to wait for. Life was beautiful.</p>
<p>But... I did realize, <em>maaybe</em> I need a way to roll back a bad deployment. <em>Maaaybe</em> I need some version control. <em>Maaaaaybe</em> it's not a good idea to have the source code living only on my laptop. <em>Maaaaaaaaybe</em> I need the build process to be reproducible. I mean, <em>come on man, I can do better</em>.</p>
<img style="display: block; margin: 0 auto; width: 500px" src="/images/was_not_that_bad.png" alt="When you take a step back, you realise that we have come a very long way." title="When you take a step back, you realise that we have come a very long way."/>
<hr />
<h3 id="where-i-am-right-now-slightly-more-complex-deployment">Where I am right now, slightly more complex deployment</h3>
<p>I do a git push to <a href="https://raccoon.onyxbits.de/blog/why-not-use-github">GitLab</a>. Everything else happens automagically: CI builds the HTML pages from the Markdown files, CI builds a Docker image containing the compiled site, then pushes it to the <a href="https://docs.gitlab.com/ee/user/packages/container_registry/">GitLab Container Registry</a> and finally, SSHes into the VPS to pull and run the new image. <a href="https://caddyserver.com/">Caddy</a> on the VPS handles TLS termination and proxies traffic to the container.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>[Local Machine]
</span><span> |
</span><span> | git push origin master
</span><span> |
</span><span> v
</span><span>[GitLab.com]
</span><span> Repository stores all source: content/, templates/, sass/, themes/, config.toml
</span><span> |
</span><span> | .gitlab-ci.yml triggers pipeline automatically
</span><span> |
</span><span> ├─ Stage 1: build-and-push-image
</span><span> | ├─ docker:dind runner
</span><span> | ├─ zola build (inside Dockerfile, Stage 1: debian:bookworm-slim)
</span><span> | ├─ output copied into nginx:alpine image (Stage 2)
</span><span> | ├─ image built and tagged: registry.gitlab.com/USERNAME/proj:<commit-sha>
</span><span> | └─ image pushed to GitLab Container Registry
</span><span> |
</span><span> └─ Stage 2: deploy-to-vps
</span><span> ├─ alpine runner with openssh-client
</span><span> └─ SSH into VPS → runs /home/mlvn/deploy.sh <commit-sha>
</span><span> |
</span><span> v
</span><span>[DigitalOcean VPS — new Droplet]
</span><span> deploy.sh:
</span><span> podman pull registry.gitlab.com/USERNAME/proj:<sha>
</span><span> podman stop nskm-site (old container)
</span><span> podman run -d --name nskm-site -p 8080:80 IMAGE:<sha>
</span><span> |
</span><span> v
</span><span>[Caddy on VPS host]
</span><span> Handles TLS (Let's Encrypt, auto-renewed)
</span><span> Caddyfile: reverse_proxy localhost:8080
</span><span> |
</span><span> v
</span><span>[Visitor's browser]
</span><span> https://nskm.xyz
</span></code></pre>
<p><strong>Pros of the CI/CD approach</strong></p>
<ul>
<li>I just git push code, site deploys itself</li>
<li>Version controlled: full git history, every change is tracked and auditable</li>
<li>Rollback: any previous image tag can be redeployed in seconds</li>
<li>Immutable deploys: each deploy is a fresh container from a known image, no partial file transfers</li>
</ul>
<p><strong>Cons of the CI/CD approach</strong></p>
<ul>
<li>More moving parts, more things that can break: GitLab, container registry, Docker/Podman, Caddy, deploy script</li>
<li>Slower feedback loop: pipeline takes more minutes</li>
<li>Registry storage to manage: images accumulate (mitigated by cleanup policy)</li>
</ul>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/ci_down.webp" alt="CI is down" title="CI is down"/>
<hr />
<h2 id="gitlab-ci">GitLab CI pipeline</h2>
<p>TODO: gitlab ci file ???</p>
<h3 id="gitlab-ci-config">The GitLab CI configuration file: <code>.gitlab-ci.yml</code></h3>
<p>Nothing fancy:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">docker:24.0.5-cli</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">services</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">docker:24.0.5-dind</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">variables</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DOCKER_TLS_CERTDIR</span><span>: </span><span style="color:#d69d85;">"/certs"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DOCKER_TLS_VERIFY</span><span>: </span><span style="color:#b5cea8;">1
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DOCKER_CERT_PATH</span><span>: </span><span style="color:#d69d85;">"$DOCKER_TLS_CERTDIR/client"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DOCKER_DRIVER</span><span>: </span><span style="background-color:#282828;color:#d69d85;">overlay2</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DOCKER_BUILDKIT</span><span>: </span><span style="color:#b5cea8;">1
</span><span> </span><span style="background-color:#282828;color:#569cd6;">IMAGE_NAME</span><span>: </span><span style="background-color:#282828;color:#d69d85;">registry.gitlab.com/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">IMAGE_TAG</span><span>: </span><span style="background-color:#282828;color:#d69d85;">$CI_COMMIT_SHORT_SHA</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">stages</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">build-and-push</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">deploy</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">build-and-push-image</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">stage</span><span>: </span><span style="background-color:#282828;color:#d69d85;">build-and-push</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">before_script</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">script</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">docker build</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">--cache-from "${IMAGE_NAME}:latest"</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">--build-arg BUILDKIT_INLINE_CACHE=1</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">-t "${IMAGE_NAME}:${IMAGE_TAG}"</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">-t "${IMAGE_NAME}:latest"</span><span>
</span><span> </span><span style="color:#b5cea8;">.
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">docker push "${IMAGE_NAME}:${IMAGE_TAG}"</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">docker push "${IMAGE_NAME}:latest"</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">$CI_COMMIT_BRANCH == "master"</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">deploy-to-vps</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">stage</span><span>: </span><span style="background-color:#282828;color:#d69d85;">deploy</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">alpine:3.19</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">before_script</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">apk add --no-cache openssh-client</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">eval $(ssh-agent -s)</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">echo "$MLVN_SSH_PRIVATE_KEY" | tr -d '\r' | base64 -d | ssh-add -</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">mkdir -p ~/.ssh && chmod 700 ~/.ssh</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">echo "$MLVN_SSH_KNOWN_HOST" >> ~/.ssh/known_hosts</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">script</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ssh -p 1234 mlvn@$MLVN_IP "/home/mlvn/deploy.sh ${IMAGE_TAG}"</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">needs</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">build-and-push-image</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">$CI_COMMIT_BRANCH == "master"</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">environment</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">production</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">url</span><span>: </span><span style="background-color:#282828;color:#d69d85;">https://nskm.xyz</span><span>
</span><span>
</span></code></pre>
<h3 id="gitlab-ci-vars">Required GitLab CI/CD variables</h3>
<table><thead><tr><th>Variable</th><th>Type</th><th>Protected</th><th>Masked</th></tr></thead><tbody>
<tr><td><code>VPS_SSH_PRIVATE_KEY</code></td><td>Variable (private key content)</td><td>Yes</td><td>Yes</td></tr>
<tr><td><code>VPS_IP</code></td><td>Variable (new Droplet IP once known)</td><td>Yes</td><td>No</td></tr>
<tr><td><code>VPS_IP</code></td><td>Variable (new Droplet IP once known)</td><td>Yes</td><td>No</td></tr>
</tbody></table>
<p>All other needed variables (<code>CI_REGISTRY_USER</code>, <code>CI_REGISTRY_PASSWORD</code>, <code>CI_REGISTRY</code>, <code>CI_COMMIT_SHORT_SHA</code>, etc.) are injected automatically by GitLab.</p>
<h3 id="gitlab-registry">GitLab container registry</h3>
<p>Every GitLab project has a <a href="https://docs.gitlab.com/user/packages/container_registry/">registry</a> at <code>registry.gitlab.com/USERNAME/PROJECT</code>. This allows me to store, manage, and distribute my <a href="https://opencontainers.org/">OCI</a> images, alongside the source code and the CI/CD pipelines. The authentication from the pipeline to the registry is automatic: GitLab injects <code>CI_REGISTRY_USER</code>, <code>CI_REGISTRY_PASSWORD</code>, <code>CI_REGISTRY</code> environment variables into every pipeline.</p>
<h4 id="vps-authentication-to-pull-images">VPS authentication to pull images</h4>
<p>The VPS serving my website need to be able to pull the container image to run it. And for being able to pull the image, it also need to authenticate itself to the container registry. This is where Gitlab deploy token enters the play.</p>
<p>A <a href="https://docs.gitlab.com/user/project/deploy_tokens/">GitLab deploy token</a> is a non-user specific credential used to authenticate automated tasks, such as <em>a VPS</em>, to access GitLab resources <em>like registries</em>.</p>
<p>I created one GitLab deploy token, I picked the <code>read_registry</code> scope, I stored the credentials in <code>~/.config/containers/auth.json</code>. This file stores your base64 encoded credentials to facilitate future registry operation.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>mlvn@mlvn:~$ cat ~/.config/containers/auth.json
</span><span>{
</span><span> </span><span style="color:#d69d85;">"auths"</span><span>: {
</span><span> </span><span style="color:#d69d85;">"registry.gitlab.com"</span><span>: {
</span><span> </span><span style="color:#d69d85;">"auth"</span><span>: </span><span style="color:#d69d85;">"XZwczpnbGR0LUdQeEg2M3pjeWlcGxveS10nskmb2tlbi1tbHZuLo2QnBGeXo1eDR0Z2l0bGFiK2"
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<h4 id="container-registry-cleanup-policy">Container registry cleanup policy</h4>
<p><a href="https://docs.gitlab.com/user/packages/container_registry/reduce_container_registry_storage/">Container registries</a> can grow in size if you don't manage your registry usage. Retrieving the list of available tags or images becomes slower. Images could take up a large amount of storage space on the server. So I defined a cleanup policy to keep last 5 tags, run weekly, remove tags older than 14 days.</p>
<h3 id="notification">Notification mechanism:</h3>
<p>I decided, GitLab CI will SSHes directly into VPS. There was two other options. The first one being a webhook receiver on VPS, which required installing & managing a webhook daemon and opening an extra port. The second one being polling. Why ?</p>
<ul>
<li>No extra service to install on VPS</li>
<li>No extra ports to open</li>
<li>Deploy action is visible in GitLab pipeline logs</li>
<li>SSH is already used and understood</li>
<li>No polling delay</li>
</ul>
<p>I generated a dedicated ED25519 key pair for GitLab CI, different from my personal key. I stored the private key in GitLab CI variable marked <code>Protected</code> + <code>Masked</code>.</p>
<hr />
<h2 id="container-image">The Container image</h2>
<h3 id="dockerfile">The multi-stage Container file</h3>
<pre data-lang="dockerfile" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-dockerfile "><code class="language-dockerfile" data-lang="dockerfile"><span style="color:#608b4e;"># Stage 1: Build static site with Zola
</span><span>
</span><span style="color:#569cd6;">FROM</span><span> docker.io/library/debian:bookworm-slim </span><span style="color:#569cd6;">AS </span><span>builder
</span><span>
</span><span style="color:#569cd6;">ARG </span><span>ZOLA_VERSION=0.18.0
</span><span style="color:#569cd6;">RUN </span><span>apt-get update && apt-get install -y --no-install-recommends \
</span><span> wget ca-certificates \
</span><span> jpegoptim optipng gifsicle libimage-exiftool-perl \
</span><span> && rm -rf /var/lib/apt/lists/*
</span><span>
</span><span style="color:#569cd6;">RUN </span><span>wget -q </span><span style="color:#d69d85;">"https://github.com/getzola/zola/releases/download/v${ZOLA_VERSION}/zola-v${ZOLA_VERSION}-x86_64-unknown-linux-gnu.tar.gz"</span><span> \
</span><span> -O /tmp/zola.tar.gz \
</span><span> && tar -xzf /tmp/zola.tar.gz -C /usr/local/bin/ \
</span><span> && chmod +x /usr/local/bin/zola \
</span><span> && rm /tmp/zola.tar.gz
</span><span>
</span><span style="color:#569cd6;">WORKDIR </span><span>/site
</span><span style="color:#569cd6;">COPY</span><span> config.toml .
</span><span style="color:#569cd6;">COPY</span><span> themes/ themes/
</span><span style="color:#569cd6;">COPY</span><span> static/ static/
</span><span style="color:#569cd6;">COPY</span><span> templates/ templates/
</span><span style="color:#569cd6;">COPY</span><span> content/ content/
</span><span>
</span><span style="color:#569cd6;">RUN </span><span>zola build \
</span><span> && find public -name </span><span style="color:#d69d85;">"*.xml"</span><span> -o -name </span><span style="color:#d69d85;">"*.html"</span><span> | xargs sed -i -e </span><span style="color:#d69d85;">'/</span><span style="color:#b4cea8;">\&</span><span style="color:#d69d85;">lt;!--.*--</span><span style="color:#b4cea8;">\&</span><span style="color:#d69d85;">gt;/d'
</span><span>
</span><span style="color:#608b4e;"># Strip metadata (EXIF, GPS, IPTC, ICC profiles) from all images
</span><span style="color:#569cd6;">RUN </span><span>exiftool -all= -overwrite_original -recurse public
</span><span>
</span><span style="color:#608b4e;"># Optimize images (lossless)
</span><span style="color:#608b4e;"># For lossy JPEG (visually identical, ~60-80% smaller): add --max=85
</span><span style="color:#569cd6;">RUN </span><span>find public -type f </span><span style="color:#b4cea8;">\(</span><span> -name </span><span style="color:#d69d85;">"*.jpg"</span><span> -o -name </span><span style="color:#d69d85;">"*.jpeg" </span><span style="color:#b4cea8;">\)</span><span> -print0 | xargs -0 -r jpegoptim --strip-all \
</span><span> && find public -type f -name </span><span style="color:#d69d85;">"*.png"</span><span> -print0 | xargs -0 -r optipng -o2 -strip all \
</span><span> && find public -type f -name </span><span style="color:#d69d85;">"*.gif"</span><span> -print0 | xargs -0 -r gifsicle -O3 --batch
</span><span>
</span><span>
</span><span style="color:#608b4e;"># Stage 2: Serve with minimal Nginx
</span><span>
</span><span style="color:#569cd6;">FROM</span><span> docker.io/library/nginx:1.25-alpine </span><span style="color:#569cd6;">AS </span><span>serve
</span><span>
</span><span style="color:#569cd6;">RUN </span><span>rm -rf /usr/share/nginx/html/*
</span><span style="color:#569cd6;">COPY</span><span> --from=builder /site/public/ /usr/share/nginx/html/
</span><span style="color:#569cd6;">COPY</span><span> nginx.conf /etc/nginx/conf.d/default.conf
</span><span>
</span><span style="color:#569cd6;">EXPOSE </span><span>80
</span><span style="color:#569cd6;">CMD </span><span>[</span><span style="color:#d69d85;">"nginx"</span><span>, </span><span style="color:#d69d85;">"-g"</span><span>, </span><span style="color:#d69d85;">"daemon off;"</span><span>]
</span></code></pre>
<h3 id="nginx-config">The Nginx configuration file</h3>
<p>Why using Nginx inside the container and not Caddy ? Nginx inside the container serves HTTP only on port 80. SSL is handled by Caddy on the VPS host outside the container. Caddy's <a href="https://caddyserver.com/features">killer features</a>, <em>you might want to sit down for this</em>, are completely wasted inside a container that only serves static files over plain HTTP to localhost. <a href="https://hub.docker.com/_/nginx">Nginx Alpine</a> is very small and purpose-built for this job.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>nsukami@IPD-DDSI5:~/GIT/nskm2$ podman images </span><span style="color:#569cd6;">| </span><span>grep nginx
</span><span>docker.io/library/nginx 1.29.5-alpine-slim a68d27696acb 3 weeks ago 13.5 MB
</span><span>docker.io/library/nginx 1.25-alpine 501d84f5d064 22 months ago 50.1 MB
</span></code></pre>
<pre data-lang="nginx" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-nginx "><code class="language-nginx" data-lang="nginx"><span>server {
</span><span> listen 80;
</span><span> root /usr/share/nginx/html;
</span><span> index index.html;
</span><span>
</span><span> gzip on;
</span><span> gzip_types text/plain text/css application/json application/javascript
</span><span> text/xml application/xml text/javascript image/svg+xml;
</span><span>
</span><span> location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff2|webp|mp4|pdf)$ {
</span><span> expires 1y;
</span><span> add_header Cache-Control "public, immutable";
</span><span> }
</span><span>
</span><span> location / {
</span><span> try_files $uri $uri/ $uri.html =404;
</span><span> }
</span><span>
</span><span> error_page 404 /404.html;
</span><span>}
</span></code></pre>
<h2 id="vps-provisioning">The VPS provisioning</h2>
<h3 id="droplet">Droplet recommendation</h3>
<p>I am a proud DigitalOcean user since 2011. At that time, I was looking for ways to learn about a real server with a real IP address, the one that are publicly accessible. At that time, I had no credit card, <em>no money to be honest</em>. My first server on this platform was offered to me by a very good friend. He paid for the VPS and gave me the root credentials, then, asked me to correctly harden the server, the rest is history.</p>
<p>For this experiment, 1 vCPU, 1GB RAM, 25GB SSD, Ubuntu 24.04 LTS should be more than enough. Keep the old Droplet running (rsync deploys still work) while you set up the new one, update DNS to the new IP once it's ready, then decommission the old one.</p>
<h3 id="provisioning-steps">Provisioning steps (in order)</h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># 1. Update packages
</span><span>sudo apt-get update </span><span style="color:#569cd6;">&& </span><span>sudo apt-get upgrade -y
</span><span>
</span><span style="color:#608b4e;"># 2. Firewall
</span><span>sudo apt-get install -y ufw
</span><span>sudo ufw allow 1234/tcp </span><span style="color:#608b4e;"># SSH on custom port
</span><span>sudo ufw allow 80/tcp </span><span style="color:#608b4e;"># HTTP (Caddy ACME challenge + redirect)
</span><span>sudo ufw allow 443/tcp </span><span style="color:#608b4e;"># HTTPS
</span><span>sudo ufw enable
</span><span>
</span><span style="color:#608b4e;"># 3. Install Podman
</span><span>sudo apt-get -y install podman
</span><span>
</span><span style="color:#608b4e;"># 4. Install Caddy
</span><span>sudo apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
</span><span>curl -1sLf </span><span style="color:#d69d85;">'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' </span><span>\
</span><span> </span><span style="color:#569cd6;">| </span><span>sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
</span><span>curl -1sLf </span><span style="color:#d69d85;">'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' </span><span>\
</span><span> </span><span style="color:#569cd6;">| </span><span>sudo tee /etc/apt/sources.list.d/caddy-stable.list
</span><span>sudo apt-get update </span><span style="color:#569cd6;">&& </span><span>sudo apt-get install -y caddy
</span><span>
</span><span style="color:#608b4e;"># 5. Configure Caddy
</span><span style="color:#608b4e;"># edit /etc/caddy/Caddyfile with the following content:
</span><span style="color:#608b4e;"># nskm.xyz {
</span><span style="color:#608b4e;"># reverse_proxy localhost:8080
</span><span style="color:#608b4e;"># }
</span><span>
</span><span style="color:#608b4e;"># 6. enable and start the service
</span><span>sudo systemctl enable caddy </span><span style="color:#569cd6;">&& </span><span>sudo systemctl start caddy
</span><span>
</span><span style="color:#608b4e;"># 7. Authenticate Podman to GitLab registry
</span><span>podman login registry.gitlab.com -u DEPLOY_TOKEN_USER -p DEPLOY_TOKEN_VALUE
</span></code></pre>
<h3 id="vps-hardening">VPS security hardening</h3>
<p>There are <a href="https://www.digitalocean.com/community/tutorials?q=%5Bsecurity%5D+harden">lots of tutorials</a> explaining how you should harden the security of your server. I think, there is no need to discuss the matter here.</p>
<hr />
<h3 id="doctl">Going further: one-command provisioning with doctl</h3>
<p><a href="https://docs.digitalocean.com/reference/doctl/">doctl</a> is DigitalOcean's CLI tool that I've never used. <a href="https://cloud-init.io/">Cloud-init</a> is a standard way to run scripts on first boot that I've never used too. I absolutely must find a way to use both tools more often.</p>
<p><strong>A cloud-init script</strong></p>
<p>It is essentially the provisioning steps from above, packed into a single file that runs automatically when the Droplet first boots. Create a file called <code>cloud-init.sh</code>:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;">#cloud-config
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">users</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">mlvn</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">groups</span><span>: </span><span style="background-color:#282828;color:#d69d85;">sudo</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">shell</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/bin/bash</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">sudo</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ALL=(ALL) NOPASSWD:ALL</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ssh_authorized_keys</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ssh-ed25519 AAAA... your-public-key-here</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">package_update</span><span>: </span><span style="color:#569cd6;">true
</span><span style="background-color:#282828;color:#569cd6;">package_upgrade</span><span>: </span><span style="color:#569cd6;">true
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">packages</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ufw</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">podman</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">fail2ban</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">unattended-upgrades</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">debian-keyring</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">debian-archive-keyring</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">apt-transport-https</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">write_files</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/etc/caddy/Caddyfile</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">content</span><span>: </span><span style="color:#569cd6;">|
</span><span style="background-color:#282828;color:#d69d85;"> nskm.xyz {
</span><span style="background-color:#282828;color:#d69d85;"> reverse_proxy localhost:8080
</span><span style="background-color:#282828;color:#d69d85;"> }
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/etc/fail2ban/jail.d/sshd.conf</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">content</span><span>: </span><span style="color:#569cd6;">|
</span><span style="background-color:#282828;color:#d69d85;"> [sshd]
</span><span style="background-color:#282828;color:#d69d85;"> enabled = true
</span><span style="background-color:#282828;color:#d69d85;"> port = 1234
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/etc/ssh/sshd_config.d/hardening.conf</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">content</span><span>: </span><span style="color:#569cd6;">|
</span><span style="background-color:#282828;color:#d69d85;"> PermitRootLogin no
</span><span style="background-color:#282828;color:#d69d85;"> PasswordAuthentication no
</span><span style="background-color:#282828;color:#d69d85;"> Port 1234
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/home/mlvn/deploy.sh</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">permissions</span><span>: </span><span style="color:#d69d85;">"0755"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">owner</span><span>: </span><span style="background-color:#282828;color:#d69d85;">mlvn:mlvn</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">content</span><span>: </span><span style="color:#569cd6;">|
</span><span style="background-color:#282828;color:#d69d85;"> #!/bin/bash
</span><span style="background-color:#282828;color:#d69d85;"> set -euo pipefail
</span><span style="background-color:#282828;color:#d69d85;"> IMAGE="registry.gitlab.com/USERNAME/proj"
</span><span style="background-color:#282828;color:#d69d85;"> TAG="${1:-latest}"
</span><span style="background-color:#282828;color:#d69d85;"> CONTAINER="nskm-site"
</span><span style="background-color:#282828;color:#d69d85;"> PORT=8080
</span><span style="background-color:#282828;color:#d69d85;"> echo "Deploying ${IMAGE}:${TAG}"
</span><span style="background-color:#282828;color:#d69d85;"> docker pull "${IMAGE}:${TAG}"
</span><span style="background-color:#282828;color:#d69d85;"> docker stop "${CONTAINER}" 2>/dev/null || true
</span><span style="background-color:#282828;color:#d69d85;"> docker rm "${CONTAINER}" 2>/dev/null || true
</span><span style="background-color:#282828;color:#d69d85;"> docker run -d --name "${CONTAINER}" --restart unless-stopped -p "${PORT}:80" "${IMAGE}:${TAG}"
</span><span style="background-color:#282828;color:#d69d85;"> docker image prune -f
</span><span style="background-color:#282828;color:#d69d85;"> echo "Done: ${IMAGE}:${TAG}"
</span><span style="background-color:#282828;color:#d69d85;">
</span><span style="background-color:#282828;color:#569cd6;">runcmd</span><span>:
</span><span> </span><span style="color:#608b4e;"># Firewall
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ufw allow 1234/tcp</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ufw allow 80/tcp</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ufw allow 443/tcp</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ufw --force enable</span><span>
</span><span>
</span><span> </span><span style="color:#608b4e;"># Install Podman
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">apt-get update && apt-get install -y podman</span><span>
</span><span>
</span><span> </span><span style="color:#608b4e;"># Install Caddy
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">apt-get update && apt-get install -y caddy</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">systemctl enable caddy && systemctl start caddy</span><span>
</span><span>
</span><span> </span><span style="color:#608b4e;"># Security
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">systemctl enable fail2ban --now</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">systemctl restart ssh</span><span>
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">dpkg-reconfigure --priority=low unattended-upgrades</span><span>
</span><span>
</span><span> </span><span style="color:#608b4e;"># Enable linger for rootless containers (if using Podman later)
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">loginctl enable-linger mlvn</span><span>
</span></code></pre>
<p><strong>Create the Droplet using the doctl command</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>doctl compute droplet create nskm-vps \
</span><span> --image ubuntu-24-04-x64 \
</span><span> --size s-1vcpu-1gb \
</span><span> --region ams3 \
</span><span> --ssh-keys $(doctl compute ssh-key list --format ID --no-header </span><span style="color:#569cd6;">| </span><span>head -1) \
</span><span> --user-data-file ./cloud-init.sh \
</span><span> --wait
</span></code></pre>
<p>And voilà!</p>
<hr />
<h3 id="deploy-script">Deploy script: <code>/home/mlvn/deploy.sh</code></h3>
<p>This is the script that will be called by Gitlab whenever a new image is built:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash
</span><span>set -euo pipefail
</span><span>
</span><span>IMAGE=</span><span style="background-color:#282828;color:#d69d85;">"registry.gitlab.com/USERNAME/nskm2"</span><span>
</span><span>TAG=</span><span style="background-color:#282828;color:#d69d85;">"${</span><span style="background-color:#282828;color:#dcdcdc;">1:-</span><span style="background-color:#282828;color:#d69d85;">latest}"</span><span>
</span><span>CONTAINER=</span><span style="background-color:#282828;color:#d69d85;">"nskm-site"</span><span>
</span><span>PORT=</span><span style="background-color:#282828;color:#d69d85;">8080</span><span>
</span><span>
</span><span>echo </span><span style="color:#d69d85;">"Deploying ${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">}"
</span><span>docker pull </span><span style="color:#d69d85;">"${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">}"
</span><span>docker stop </span><span style="color:#d69d85;">"${</span><span>CONTAINER</span><span style="color:#d69d85;">}" </span><span style="color:#b5cea8;">2</span><span>>/dev/null </span><span style="color:#569cd6;">|| </span><span>true
</span><span>docker rm </span><span style="color:#d69d85;">"${</span><span>CONTAINER</span><span style="color:#d69d85;">}" </span><span style="color:#b5cea8;">2</span><span>>/dev/null </span><span style="color:#569cd6;">|| </span><span>true
</span><span>docker run -d --name </span><span style="color:#d69d85;">"${</span><span>CONTAINER</span><span style="color:#d69d85;">}"</span><span> --restart unless-stopped -p </span><span style="color:#d69d85;">"${</span><span>PORT</span><span style="color:#d69d85;">}:80" "${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">}"
</span><span>docker image prune -f
</span><span>echo </span><span style="color:#d69d85;">"Done: ${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">}"
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>chmod +x /home/mlvn/deploy.sh
</span></code></pre>
<h3 id="podman-critical">Critical: rootless Podman and SSH session lifecycle</h3>
<p>During my various tests, I noticed some strange behaviour. The deploy log shows success and the health check passes, but the site returns 502 immediately after the CI job finishes:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>$ # Job succeeded....
</span><span>$ ssh -p 1234 mlvn@$MLVN_IP "/home/mlvn/deploy.sh ${IMAGE_TAG}"
</span><span>Deploying registry.gitlab.com/nskm/mlvn:b7fg2abf
</span><span>...
</span><span>Running health check...
</span><span>Done: registry.gitlab.com/nskm/mlvn:b7fg2abf is live
</span><span>Job succeeded
</span><span>
</span><span>$ # Yet...
</span><span>$ curl -I "http://1.2.3.4"
</span><span>HTTP/1.1 502 Bad Gateway
</span><span>Server: Caddy
</span></code></pre>
<p>Even more confusing...</p>
<p><code>podman ps -a</code> shows the container in <code>Exited (0)</code> state, and <code>podman logs nskm-site</code> confirms it received SIGTERM seconds after the health check passed:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>22:03:54 — container starts
</span><span>22:03:57 — health check hits it: "GET / HTTP/1.1" 200 ✓
</span><span>22:04:07 — signal 15 (SIGTERM) received, exiting ← WTF?!
</span></code></pre>
<br>
<img style="display: block; margin: 0 auto; width: 300px" src="/images/kanye_wtf.gif" alt="wtf" title="wtf"/>
<br>
<p><a href="https://lists.podman.io/archives/list/podman@lists.podman.io/thread/CQTZAORZXW6UWDNAV5TAWAV6PTDJFW6V/">The problem</a>: When using rootless Podman (Podman running as a regular user, not root), containers are tied to the user's login session. When a CI job SSHes into the VPS, runs <code>deploy.sh</code>, and disconnects, the SSH session ends and systemd cleans up the session, sending SIGTERM to all processes in it, including the running container.</p>
<p><a href="https://www.freedesktop.org/software/systemd/man/latest/loginctl.html">The solution</a>: enabling lingering for the user running the container:**</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>loginctl enable-linger mlvn </span><span style="color:#608b4e;"># run once on the VPS
</span></code></pre>
<p>This instructs systemd to keep the <code>mlvn</code> user session alive permanently, even after logout. Containers started by this user will survive SSH disconnection.</p>
<p><strong>Verify:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>loginctl show-user mlvn </span><span style="color:#569cd6;">| </span><span>grep Linger
</span><span style="color:#608b4e;"># Expected output: Linger=yes
</span></code></pre>
<p><strong>This must be run before the first real deploy.</strong> Without it, every CI deploy will appear to succeed but the site will go down immediately after the SSH session closes.</p>
<hr />
<h2 id="ssl-tls">SSL & Let's Encrypt</h2>
<h3 id="caddy">Decision: Caddy</h3>
<p>Caddy handles everything automatically with a single config line:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>nskm.xyz {
</span><span> reverse_proxy localhost:8080
</span><span>}
</span></code></pre>
<p>In 3 lines, Caddy:</p>
<ul>
<li>Obtains Let's Encrypt cert on first request automatically</li>
<li>Renews before expiry (auto, no cron needed, no certbot)</li>
<li>Redirects HTTP → HTTPS automatically</li>
<li>Stores certs in <code>/var/lib/caddy/.local/share/caddy/</code></li>
</ul>
<p>I remember the time when I had to <em>get my geek on</em> and do eveything myself, <em>as a proud Linux user</em>: certbot install, cron job, renewal hooks, manual intervention when things break. <em>Phew</em>!</p>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/jpoesen.png" alt="A very looooooong time ago" title="A very looooooong time ago"/>
<br>
<h3 id="letsencrypt-ip">Let's Encrypt IP address certificates (January 2026)</h3>
<p>Amazing how few years ago, it was something you didn't think about. Today, Let's Encrypt now issues <a href="https://letsencrypt.org/2026/01/15/6day-and-ip-general-availability">certificates for raw IP addresses</a>. I was thinking about using it for the new VPS, before updating the DNS to the new IP address. Then I learned IP address certs are <a href="https://letsencrypt.org/2026/01/15/6day-and-ip-general-availability">short-lived certificates</a> — they are valid only for <strong>160 hours (≈6 days)</strong> instead of the usual 90 days. Not a problem in practice, just worth knowing.</p>
<p>Where this could be useful on this VPS ? Maybe ....</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>YOUR_VPS_IP {
</span><span> reverse_proxy localhost:SOME_PORT
</span><span>}
</span></code></pre>
<hr />
<h2 id="operations">Operations</h2>
<h3 id="rollback">Rollback procedure</h3>
<p>One of the reasons <em>(probably the main reason)</em> I moved to CI/CD was the ability to rollback a deployment. With rsync, rolling back meant "do you still have the old files somewhere?". With container images, every deploy is tagged with its commit SHA and stored in the GitLab registry.</p>
<p><strong>Available image tags on the VPS:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>mlvn@mlvn:~$ podman login registry.gitlab.com
</span><span>Authenticating with existing credentials for registry.gitlab.com
</span><span>Existing credentials are valid. Already logged in to registry.gitlab.com
</span><span>mlvn@mlvn:~$
</span><span>mlvn@mlvn:~$ podman images --format </span><span style="color:#d69d85;">"{{.Repository}}:{{.Tag}} {{.CreatedAt}}" </span><span style="color:#569cd6;">| </span><span>grep nskm
</span><span>registry.gitlab.com/nskm/mlvn:6c08ce72 2026-03-03 06:59:24 +0000 UTC
</span><span>registry.gitlab.com/nskm/mlvn:latest 2026-03-01 21:46:30 +0000 UTC
</span><span>registry.gitlab.com/nskm/mlvn:b7fb2abf 2026-03-01 21:46:30 +0000 UTC
</span><span>registry.gitlab.com/nskm/mlvn:9a326e8c 2026-03-01 21:30:53 +0000 UTC
</span><span>registry.gitlab.com/nskm/mlvn:005ba0d9 2026-03-01 17:21:40 +0000 UTC
</span><span>mlvn@mlvn:~$
</span></code></pre>
<p><strong>Roll back to a previous version:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>mlvn@mlvn:~$ ./deploy.sh b7fb2abf
</span><span>Login in to the Gitlab container registry
</span><span>Authenticating with existing credentials for registry.gitlab.com
</span><span>Existing credentials are valid. Already logged in to registry.gitlab.com
</span><span>Deploying registry.gitlab.com/nskm/mlvn:b7fb2abf
</span><span>Trying to pull registry.gitlab.com/nskm/mlvn:b7fb2abf...
</span><span>Getting image source signatures
</span><span>Copying blob 5406ed7b06d9 skipped: already exists
</span><span>Copying blob 4abcf2066143 skipped: already exists
</span><span>Copying blob fc21a1d387f5 skipped: already exists
</span><span>Copying blob e6ef242c1570 skipped: already exists
</span><span>Copying blob 13fcfbc94648 skipped: already exists
</span><span>Copying blob d4bca490e609 skipped: already exists
</span><span>Copying blob 8a3742a9529d skipped: already exists
</span><span>Copying blob 0d0c16747d2c skipped: already exists
</span><span>Copying blob f7809d4c4360 skipped: already exists
</span><span>Copying blob 3ee6c312ca9b skipped: already exists
</span><span>Copying blob ed4f6e59d232 skipped: already exists
</span><span>Copying config 632bfe60dc done </span><span style="color:#569cd6;">|
</span><span>Writing manifest to image destination
</span><span>632bfe60dce5626a87db391496d017ed1dc7d5f805af81fa5936cbb0323c0099
</span><span>nskm-site
</span><span>nskm-site
</span><span>ef7a0a78ee6b047b5fbb0c5b4e6622fff4769096d12ec21384c36149d000f669
</span><span>Running health check...
</span><span>Done: registry.gitlab.com/nskm/mlvn:b7fb2abf is live
</span><span>mlvn@mlvn:~$
</span></code></pre>
<p>That's it. The deploy script pulls the old image (already cached locally if it was recently used), stops the current container, and starts a new one based on the old image. The site is back to its previous state in seconds.</p>
<p>If the image has been pruned locally, Podman pulls it from the GitLab registry; as long as it's within the cleanup policy window.</p>
<hr />
<h3 id="image-scanning">Container image security scanning</h3>
<p>Every container image inherits vulnerabilities from its base image. While <code>debian:bookworm-slim</code> and <code>nginx:1.25-alpine</code> are well-maintained, they still ship with system libraries that occasionally have known CVEs.</p>
<p><a href="https://trivy.dev/">Trivy</a> is a free, open-source scanner that checks container images against vulnerability databases. Adding it as a CI stage between build and deploy means a vulnerable image never reaches the VPS.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">scan-image</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">stage</span><span>: </span><span style="background-color:#282828;color:#d69d85;">scan</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">aquasec/trivy:latest</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">entrypoint</span><span>: [</span><span style="color:#d69d85;">""</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">script</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">trivy image</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">--exit-code 1</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">--severity HIGH,CRITICAL</span><span>
</span><span> </span><span style="background-color:#282828;color:#d69d85;">--no-progress</span><span>
</span><span> </span><span style="color:#d69d85;">"${IMAGE_NAME}:${IMAGE_TAG}"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">needs</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">build-and-push-image</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">if</span><span>: </span><span style="background-color:#282828;color:#d69d85;">$CI_COMMIT_BRANCH == "master"</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">allow_failure</span><span>: </span><span style="color:#569cd6;">true
</span></code></pre>
<p><code>--exit-code 1</code> makes the pipeline fail if HIGH or CRITICAL vulnerabilities are found. <code>allow_failure: true</code> means the deploy can still proceed — it's a warning, not a blocker. Remove <code>allow_failure</code> once you're confident in your base images.</p>
<hr />
<h3 id="monitoring">Uptime monitoring</h3>
<p>The CI pipeline can succeed and the site can still go down — I learned that the hard way with the Podman linger issue. The pipeline reports success, you go to bed, and the site has been returning 502 for hours. Nobody notices until someone tries to visit.</p>
<p><a href="https://uptimerobot.com/">UptimeRobot</a> pings your site every 5 minutes and notifies you when it's unreachable. In this real, there is also <a href="#">Hetrix</a>, <a href="#">StatusCake</a>, <a href="#">Odown</a>, <a href="#">Uptime Kuma</a>, <a href="#">Better Stack</a>. Never tried none of them.</p>
<p>This is just a way to close the <a href="https://www.dynatrace.com/news/blog/what-is-observability-2/">observability</a> gap: the CI watches the deploy process, and <em>something</em> watches the <strong>deployed result</strong>.</p>
<img style="display: block; margin: 0 auto; width: 350px" src="/images/observability.gif" alt="observing" title="Observing"/>
<p><br><br></p>
<hr />
<h3 id="zero-downtime">Zero-downtime deploys</h3>
<p><a href="https://lpalmieri.com/posts/zero-downtime-deployments/">Zero downtime deployment</a> (ZDD) is the art of updating applications without interrupting service by running <strong>new</strong> and <strong>old</strong> versions simultaneously.</p>
<p>The current deploy script has a brief downtime window: stop old container, start new one. For a personal blog, nobody notices. But it's a fun problem to solve and a useful pattern to know.</p>
<p><strong>What I did:</strong> run two containers on two ports (8080 and 8081). Caddy <a href="https://caddyserver.com/docs/caddyfile/directives/reverse_proxy">reverse proxies</a> to both. At any given time, only one is alive.</p>
<p><strong>Updated Caddyfile:</strong></p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>nskm.xyz {
</span><span> reverse_proxy localhost:8080 localhost:8081
</span><span>}
</span></code></pre>
<p>Caddy with multiple upstreams does <a href="https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#passive-health-checks">passive health checking</a>, meaning, if one upstream doesn't respond, it routes to the other. If a request hits a dead upstream, Caddy retries on the next one before returning an error.</p>
<p><strong>Updated deploy script:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash
</span><span>set -euo pipefail
</span><span>
</span><span>IMAGE=</span><span style="background-color:#282828;color:#d69d85;">"registry.gitlab.com/USERNAME/proj"</span><span>
</span><span>TAG=</span><span style="background-color:#282828;color:#d69d85;">"${</span><span style="background-color:#282828;color:#dcdcdc;">1:-</span><span style="background-color:#282828;color:#d69d85;">latest}"</span><span>
</span><span>
</span><span>podman pull </span><span style="color:#d69d85;">"${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">}"
</span><span>
</span><span style="color:#608b4e;"># Which port is currently in use?
</span><span style="color:#569cd6;">if </span><span>podman ps --format </span><span style="color:#d69d85;">"{{.Names}}" </span><span style="color:#569cd6;">| </span><span>grep -q </span><span style="color:#d69d85;">"nskm-site-8080"</span><span style="color:#569cd6;">; then
</span><span> OLD_PORT=</span><span style="background-color:#282828;color:#d69d85;">8080</span><span style="color:#569cd6;">; </span><span>NEW_PORT=</span><span style="background-color:#282828;color:#d69d85;">8081</span><span>
</span><span style="color:#569cd6;">else
</span><span> OLD_PORT=</span><span style="background-color:#282828;color:#d69d85;">8081</span><span style="color:#569cd6;">; </span><span>NEW_PORT=</span><span style="background-color:#282828;color:#d69d85;">8080</span><span>
</span><span style="color:#569cd6;">fi
</span><span>
</span><span>echo </span><span style="color:#d69d85;">"Deploying ${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">} on port ${</span><span>NEW_PORT</span><span style="color:#d69d85;">}"
</span><span>
</span><span style="color:#608b4e;"># Start new container on the free port
</span><span>podman run -d --name </span><span style="color:#d69d85;">"nskm-site-${</span><span>NEW_PORT</span><span style="color:#d69d85;">}" </span><span>\
</span><span> --restart unless-stopped -p </span><span style="color:#d69d85;">"${</span><span>NEW_PORT</span><span style="color:#d69d85;">}:80" "${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">}"
</span><span>
</span><span style="color:#608b4e;"># Health check
</span><span>sleep 2
</span><span style="color:#569cd6;">if ! </span><span>curl -sf --max-time 5 </span><span style="color:#d69d85;">"http://localhost:${</span><span>NEW_PORT</span><span style="color:#d69d85;">}" </span><span>> /dev/null</span><span style="color:#569cd6;">; then
</span><span> echo </span><span style="color:#d69d85;">"Health check FAILED — old container still live on port ${</span><span>OLD_PORT</span><span style="color:#d69d85;">}"
</span><span> podman rm -f </span><span style="color:#d69d85;">"nskm-site-${</span><span>NEW_PORT</span><span style="color:#d69d85;">}"
</span><span> exit 1
</span><span style="color:#569cd6;">fi
</span><span>
</span><span style="color:#608b4e;"># Stop old container (Caddy automatically routes to the new one)
</span><span>podman stop </span><span style="color:#d69d85;">"nskm-site-${</span><span>OLD_PORT</span><span style="color:#d69d85;">}" </span><span style="color:#b5cea8;">2</span><span>>/dev/null </span><span style="color:#569cd6;">|| </span><span>true
</span><span>podman rm </span><span style="color:#d69d85;">"nskm-site-${</span><span>OLD_PORT</span><span style="color:#d69d85;">}" </span><span style="color:#b5cea8;">2</span><span>>/dev/null </span><span style="color:#569cd6;">|| </span><span>true
</span><span>
</span><span>podman image prune -f
</span><span>echo </span><span style="color:#d69d85;">"Done: ${</span><span>IMAGE</span><span style="color:#d69d85;">}:${</span><span>TAG</span><span style="color:#d69d85;">} live on port ${</span><span>NEW_PORT</span><span style="color:#d69d85;">}"
</span></code></pre>
<p><strong>What happens during a deploy:</strong></p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>Time ──────────────────────────────────────────────────►
</span><span>
</span><span>Old container (:8080) ████████████████████████████████████▓░░░░░░░░░░
</span><span>New container (:8081) ░░░░▓████████████████████████████████
</span><span>Caddy routes to ──8080────────────────────────────┐
</span><span> └──8081───────
</span><span>Health check ✓
</span><span> ▲
</span><span> old stopped here
</span><span> zero downtime
</span></code></pre>
<p>Both containers run simultaneously during the transition. Caddy's health checking handles the routing. The deploy script only needs to know what is the port Podman is currently running on.</p>
<hr />
<h2 id="more">More on this topic</h2>
<p>The shift from manual rsync deployments to automated CI/CD took few hours of research, mistakes, and refinement. The hardest part wasn't the Docker setup or GitLab CI — it was that subtle Podman session lifecycle issue that made the site silently disappear after every successful deploy.</p>
<p>But now, deploying is automagic. My site has a full audit trail. I can rollback to any previous version in seconds. And, now that I'm looking at it, I'm no longer tethered to one computer.</p>
<p>The trade-offs are real: the pipeline takes longer and there are more moving parts. Yet, I would like to go even further, and make things even funnier: running a Pod inside the VPS, and inside the Pod, having 2 containers, one for the website, and another for, I don't know, collect & ship the logs ?! We'll see.</p>
<img style="display: block; margin: 0 auto; width: 350px" src="/images/moving_parts.gif" alt="Too many moving parts" title="Too many moving parts"/>
<p><br><br></p>
<p><strong>Here is the big picture showing what has been done:</strong></p>
<pre data-lang="text" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-text "><code class="language-text" data-lang="text"><span>+-------------+ git push +-----------------------------------------+
</span><span>| | ------------------------> | GitLab.com |
</span><span>| | | |
</span><span>| Laptop | email on failure | |
</span><span>| | <------------------------ | +-----------------------------------+ |
</span><span>+-------------+ | | CI/CD Pipeline | |
</span><span> | | | |
</span><span> | | 1. build-and-push-image | |
</span><span> | | docker build | |
</span><span> | | docker push -----------------------------+
</span><span> | | | | |
</span><span> | | 2. scan-image (Trivy) | | |
</span><span> | | checks for CVEs | | |
</span><span> | | | | |
</span><span> | | 3. deploy-to-vps | | |
</span><span> | | SSH ---------------------------------+ |
</span><span> | | | | | |
</span><span> | +-----------------------------------+ | | |
</span><span> +-----------------------------------------+ | |
</span><span> | |
</span><span> +--------------------------------------+ | |
</span><span> | GitLab Container Registry | | |
</span><span> | | | |
</span><span> | IMAGE:abc1234f | | |
</span><span> | IMAGE:def5678a | <--------+
</span><span> | IMAGE:latest | |
</span><span> | | |
</span><span> +-----------------+--------------------+ |
</span><span> | |
</span><span> podman pull |
</span><span> | |
</span><span> v |
</span><span>+--------------+ pings every 5 min +--------------------------------------+ |
</span><span>| | ----------------------> | DigitalOcean VPS | |
</span><span>| UptimeRobot | | | <----+
</span><span>| | <-- site unreachable | +--------+ +------------------+ |
</span><span>| alerts via | | | Caddy | | Nginx container | |
</span><span>| email/SMS | | | :443 +--->| :8080 | |
</span><span>+--------------+ | | | +------------------+ |
</span><span> | | TLS | +------------------+ |
</span><span> | | HTTPS +--->| Nginx container | |
</span><span> | | | | :8081 | |
</span><span> | +---+----+ +------------------+ |
</span><span> +------+-------------------------------+
</span><span> |
</span><span> HTTPS :443
</span><span> |
</span><span> +------+-------+
</span><span> | Visitor |
</span><span> | Browser |
</span><span> +--------------+
</span></code></pre>
<p>If you learned something, please <em>like, share and subscribe</em>. Here are some resources to deepen your understanding:</p>
<ul>
<li><a href="https://www.reddit.com/r/devops/comments/1p8vyxi/zero_downtime_deployments_without_kubernetes/">Zero downtime deploymets without K8s</a></li>
<li><a href="https://docs.podman.io/en/latest/">What is Podman</a></li>
<li><a href="https://docs.gitlab.com/ee/ci/">GitLab CI/CD documentation</a></li>
<li><a href="https://www.youtube.com/watch?v=yxNpIL2MqEY">We use containers now. Here is why</a></li>
<li><a href="https://certbot.eff.org/">Certbot</a></li>
<li><a href="https://caddyserver.com/docs/">Caddy documentation</a></li>
<li><a href="https://www.youtube.com/watch?v=w_lLt4lLplE">The intro to Docker I wish I had when I started</a></li>
<li><a href="https://www.docker.com/resources/what-is-container-registry/">Container registries 101</a></li>
<li><a href="https://letsencrypt.org/how-it-works/">Let's encrypt how it works</a></li>
<li><a href="https://www.youtube.com/watch?v=W3EoZpCr1O8">Playing with Caddy server</a></li>
<li><a href="https://www.getzola.org/">Zola Documentation</a></li>
<li><a href="https://status.gitlab.com/">Gitlab system status</a></li>
</ul>
Vectors2026-02-26T00:00:00+00:002026-02-26T00:00:00+00:00
Unknown
https://nskm.xyz/posts/vctrs/ <img style="display: block; margin: 0 auto; width: 400px" src="/images/vectors.png" alt="vectors" title="vectors"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol start="0">
<li>
<p><strong>Opinions</strong>: The views expressed here are my own unless otherwise specified. They do not represent any employer, organization, or affiliated group.</p>
</li>
<li>
<p><strong>Code examples</strong>: All code examples are educational. Production use requires careful consideration of performance, security, and your specific data characteristics.</p>
</li>
<li>
<p><strong>Model choice</strong>: The <code>all-MiniLM-L6-v2</code> model is a practical default, but embedding model selection should depend on your domain. Consider fine-tuning for specialized use cases.</p>
</li>
<li>
<p><strong>This article is for</strong>: Django developers curious about semantic search and vector embeddings, or anyone wanting to understand how vectors fit alongside traditional structured data.</p>
</li>
<li>
<p>**This article was written on a Framework laptop 16, <em>yeah</em>!
<br>
<br></p>
</li>
</ol>
<h2 id="hook">Hook</h2>
<p>When I start a new <a href="#">Django</a> project, the instinct is immediate and familiar: open <code>models.py</code> and start writing <a href="#">classes</a>. I think about my entity, its attributes, its relationships, and I translate that mental model almost directly into Python. That's great. It works. Since so many years. It's readable. It's what Django was built for. Amazing. Perfect. Flawless.</p>
<p>If you follow my amazing and colossal work, you know that since more that two years now, I'm working on a project in the public health sector. I'm handling lots and lots and lots of data. I have to answer interesting questions like... Yet, I'm still reprensenting my data using classes and attributes. Reflexes.</p>
<p>There's another way to represent data that's been quietly powering search engines, recommendation systems, and AI applications for years, it's increasingly accessible to application developers like me. I should learn to use it more often. Instead of describing data as a collection of named attributes, I can describe it as a <strong>point in a high-dimensional space</strong>, a <a href="https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics)">vector</a>.</p>
<p>This article walks through what that actually means, when it matters, and how to implement both approaches side by side in a real Django project. To increase your chances of understanding, it is recommended that you listen to this <a href="https://www.youtube.com/watch?v=amBBO4PqJKo">track</a> while reading the article. Enjoy!</p>
<hr />
<h2 id="toc">ToC</h2>
<ol>
<li><a href="https://nskm.xyz/posts/vctrs/#classical-approach">The classical approach</a></li>
<li><a href="https://nskm.xyz/posts/vctrs/#vector-approach">The vector approach</a></li>
<li><a href="https://nskm.xyz/posts/vctrs/#why-not-ai">Why this is not just for AI</a></li>
<li><a href="https://nskm.xyz/posts/vctrs/#both-approaches">Both approaches in Django</a></li>
<li><a href="https://nskm.xyz/posts/vctrs/#going-further">Going further</a></li>
<li><a href="https://nskm.xyz/posts/vctrs/#broader-landscape">Representation models</a></li>
<li><a href="https://nskm.xyz/posts/vctrs/#how-to-practice">How to practice</a></li>
</ol>
<p><br><br></p>
<h2 id="classical-approach">The classical approach: data as structure</h2>
<p>When you model a medical facility in a <a href="https://nskm.xyz/posts/syndromic-surveillance/">surveillance platform</a>, you probably write something like this:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">class </span><span>HealthFacility(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> name = models.CharField(max_length=</span><span style="color:#b5cea8;">255</span><span>)
</span><span> region = models.CharField(max_length=</span><span style="color:#b5cea8;">100</span><span>)
</span><span> facility_type = models.CharField(max_length=</span><span style="color:#b5cea8;">50</span><span>) </span><span style="color:#608b4e;"># hospital, clinic, health_post
</span><span> capacity = models.IntegerField()
</span><span> services = models.ManyToManyField(</span><span style="color:#d69d85;">'Service'</span><span>)
</span></code></pre>
<p>This is <strong>structured representation</strong>. Data lives in named slots. Querying it means matching those slots exactly:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># Find clinics in Dakar with pediatric services
</span><span>HealthFacility.objects.filter(
</span><span> region=</span><span style="color:#d69d85;">"Dakar"</span><span>,
</span><span> facility_type=</span><span style="color:#d69d85;">"clinic"</span><span>,
</span><span> services__name=</span><span style="color:#d69d85;">"pediatrics"
</span><span>)
</span></code></pre>
<p>This is powerful for what it is. You know exactly what you're looking for, the schema enforces consistency, and <a href="https://www.postgresql.org/docs/">SQL</a> is extraordinarily good at these kinds of exact, predicate-based lookups.</p>
<p>The limitation reveals itself the moment your questions become fuzzy. What if you want facilities that are <em>similar</em> to a given one — without being able to define exactly what "similar" means? What if a user types "district hospital with maternal care near the coast" and you need to find the best match? Predicate-based filtering breaks down because similarity isn't a predicate. It's a geometry problem.</p>
<hr />
<h2 id="vector-approach">The vector approach: data as position</h2>
<p>A vector is simply an ordered list of numbers:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>[0.82, 0.14, 0.67, 0.03, 0.91, ...]
</span></code></pre>
<p>The idea is that you can encode the <em>meaning</em> or <em>characteristics</em> of any piece of data as a point in a high-dimensional space, such that <strong>things which are semantically similar end up close together</strong>, and things that are different end up far apart.</p>
<p>This is not hand-crafted feature engineering <em>(though it can be)</em>. <a href="https://huggingface.co/spaces/mteb/leaderboard">Modern embedding models</a> — neural networks trained on massive datasets — can take a sentence, a document, even an image, and produce a dense vector that captures its semantic content. The model has learned, from context, what concepts relate to each other. "Maternal health clinic" and "obstetric care center" will produce vectors that are very close. "Warehouse" will be far away.</p>
<p>The distance between two vectors is typically measured with <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a>:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>similarity = cos(θ) = (A · B) / (||A|| × ||B||)
</span></code></pre>
<p>A value of 1 means identical direction <em>(semantically equivalent)</em>. A value of 0 means orthogonal <em>(unrelated)</em>. This single number replaces the entire predicate matching logic.</p>
<hr />
<h2 id="why-not-ai">Why this is not just for AI projects</h2>
<p>You might think: "I don't need semantic search. I need to store patient records." That's fair. This is exactly how I've done it on the <a href="https://rh.pasteur.sn/en/research-and-public-health/epidemiology-clinical-research-and-data-science/fever-surveillance-syndromic-sentinel-surveillance-senegal-4s-network">4S project</a>. But I'm more and more facing these new types of scenarios:</p>
<p><strong>Duplicate detection.</strong> Two users register "Hôpital Principal de Dakar" and "hopital principal dakar". Exact matching fails. Vector similarity catches them as near-duplicates.</p>
<p><strong>Recommendation.</strong> "Users who viewed this report also found these relevant." You don't need a collaborative filtering setup — you can just find the reports whose vectors are closest to the one currently being read.</p>
<p><strong>Free-text search that actually works.</strong> Users don't search with your field names. They type natural language. Vectors let you match that intent.... <em>This one stung me very badly recently.</em></p>
<p><strong>Clustering without labels.</strong> You have 10,000 epidemiological case reports. You want to discover natural groupings without writing filtering rules. Find neighborhoods in vector space.... We are working in this particular topic. I look at how we are implementing this feature and I am already disgusted by the direction we are taking. <em>Marvelous</em>.</p>
<p><strong>Anomaly detection.</strong> A new weekly report has a vector that is very far from all historical ones. Flag it for review.</p>
<hr />
<h2 id="both-approaches">Both approaches in Django: a concrete example</h2>
<p>Let's build a very small project that models epidemiological reports using both approaches. We'll use <a href="https://github.com/pgvector/pgvector">pgvector</a>, a <a href="https://www.postgresql.org/docs/current/extend.html">PostgreSQL extension</a>, so vectors live right next to our traditional Django models. No separate <a href="https://en.wikipedia.org/wiki/Vector_database">vector databases</a>, no need for new infrastructure if you're already on the amazing <a href="https://www.postgresql.org/">Postgres</a>.</p>
<h3 id="setup">Setup</h3>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>pip install django pgvector sentence-transformers psycopg2-binary </span><span style="color:#608b4e;"># easy peasy
</span></code></pre>
<p>Enable the extension in a Django migration:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">from </span><span>pgvector.django </span><span style="color:#9b9b9b;">import </span><span>VectorExtension
</span><span>
</span><span style="color:#569cd6;">class </span><span>Migration(</span><span style="color:#4ec9b0;">migrations.Migration</span><span>):
</span><span> operations = [
</span><span> VectorExtension(),
</span><span> ]
</span></code></pre>
<h3 id="the-traditional-model">The traditional model</h3>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># surveillance/models.py
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>django.db </span><span style="color:#9b9b9b;">import </span><span>models
</span><span style="color:#9b9b9b;">from </span><span>pgvector.django </span><span style="color:#9b9b9b;">import </span><span>VectorField
</span><span>
</span><span style="color:#569cd6;">class </span><span>EpiReport(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> </span><span style="color:#608b4e;">"""
</span><span style="color:#608b4e;"> Classical structured representation.
</span><span style="color:#608b4e;"> Every piece of information lives in a named, typed field.
</span><span style="color:#608b4e;"> """
</span><span> title = models.CharField(max_length=</span><span style="color:#b5cea8;">255</span><span>)
</span><span> region = models.CharField(max_length=</span><span style="color:#b5cea8;">100</span><span>)
</span><span> disease = models.CharField(max_length=</span><span style="color:#b5cea8;">100</span><span>)
</span><span> week = models.IntegerField()
</span><span> year = models.IntegerField()
</span><span> case_count = models.IntegerField(default=</span><span style="color:#b5cea8;">0</span><span>)
</span><span> severity = models.CharField(
</span><span> max_length=</span><span style="color:#b5cea8;">20</span><span>,
</span><span> choices=[(</span><span style="color:#d69d85;">"low"</span><span>, </span><span style="color:#d69d85;">"Low"</span><span>), (</span><span style="color:#d69d85;">"moderate"</span><span>, </span><span style="color:#d69d85;">"Moderate"</span><span>), (</span><span style="color:#d69d85;">"high"</span><span>, </span><span style="color:#d69d85;">"High"</span><span>)]
</span><span> )
</span><span> summary = models.TextField()
</span><span>
</span><span> </span><span style="color:#608b4e;"># The vector representation lives alongside the structured one.
</span><span> </span><span style="color:#608b4e;"># 384 dimensions is the output size of the model we'll use.
</span><span> embedding = VectorField(dimensions=</span><span style="color:#b5cea8;">384</span><span>, null=</span><span style="color:#569cd6;">True</span><span>, blank=</span><span style="color:#569cd6;">True</span><span>)
</span><span>
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> indexes = [
</span><span> </span><span style="color:#608b4e;"># Traditional index for structured queries
</span><span> models.Index(fields=[</span><span style="color:#d69d85;">"region"</span><span>, </span><span style="color:#d69d85;">"disease"</span><span>, </span><span style="color:#d69d85;">"year"</span><span>]),
</span><span> ]
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>__str__(self):
</span><span> </span><span style="color:#569cd6;">return f</span><span style="color:#d69d85;">"</span><span>{self.title}</span><span style="color:#d69d85;"> (</span><span>{self.region}</span><span style="color:#d69d85;">, W</span><span>{self.week}</span><span style="color:#d69d85;">/</span><span>{self.year}</span><span style="color:#d69d85;">)"
</span></code></pre>
<p>Note that both representations coexist in the same table. This is intentional — we're not replacing one with the other. We're giving ourselves two different query interfaces into the same data.</p>
<h3 id="generating-the-embedding">Generating the embedding</h3>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># surveillance/embeddings.py
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>sentence_transformers </span><span style="color:#9b9b9b;">import </span><span>SentenceTransformer
</span><span>
</span><span style="color:#608b4e;"># All-MiniLM-L6-v2 is small (80MB), fast, and produces 384-dim vectors.
</span><span style="color:#608b4e;"># It runs on CPU without issue, which matters in resource-constrained environments.
</span><span style="color:#608b4e;"># See here: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
</span><span>
</span><span>_model = </span><span style="color:#569cd6;">None
</span><span>
</span><span style="color:#569cd6;">def </span><span>get_model():
</span><span> </span><span style="color:#569cd6;">global </span><span>_model
</span><span> </span><span style="color:#569cd6;">if </span><span>_model </span><span style="color:#569cd6;">is None</span><span>:
</span><span> _model = SentenceTransformer(</span><span style="color:#d69d85;">"all-MiniLM-L6-v2"</span><span>)
</span><span> </span><span style="color:#569cd6;">return </span><span>_model
</span><span>
</span><span style="color:#569cd6;">def </span><span>embed_report(report: </span><span style="color:#d69d85;">"EpiReport"</span><span>) -> list[float]:
</span><span> </span><span style="color:#608b4e;">"""
</span><span style="color:#608b4e;"> Convert a report's semantic content into a vector.
</span><span style="color:#608b4e;"> The text we embed is a natural language description of the report,
</span><span style="color:#608b4e;"> so the model can capture the meaning, not just the keywords.
</span><span style="color:#608b4e;"> """
</span><span> text = (
</span><span> </span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"</span><span>{report.title}</span><span style="color:#d69d85;">. "
</span><span> </span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"Disease: </span><span>{report.disease}</span><span style="color:#d69d85;">. "
</span><span> </span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"Region: </span><span>{report.region}</span><span style="color:#d69d85;">. "
</span><span> </span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"Severity: </span><span>{report.severity}</span><span style="color:#d69d85;">. "
</span><span> </span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"</span><span>{report.summary}</span><span style="color:#d69d85;">"
</span><span> )
</span><span> model = get_model()
</span><span> </span><span style="color:#569cd6;">return </span><span>model.encode(text).tolist()
</span></code></pre>
<h3 id="the-two-query-styles-side-by-side">The two query styles, side by side</h3>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># surveillance/queries.py
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>pgvector.django </span><span style="color:#9b9b9b;">import </span><span>CosineDistance
</span><span style="color:#9b9b9b;">from .</span><span>models </span><span style="color:#9b9b9b;">import </span><span>EpiReport
</span><span style="color:#9b9b9b;">from .</span><span>embeddings </span><span style="color:#9b9b9b;">import </span><span>get_model
</span><span>
</span><span style="color:#608b4e;"># ── Approach 1: Structured Query ────────────────────────────────────────────
</span><span>
</span><span style="color:#569cd6;">def </span><span>get_high_severity_reports(region: str, disease: str, year: int):
</span><span> </span><span style="color:#608b4e;">"""
</span><span style="color:#608b4e;"> Classical Django ORM query.
</span><span style="color:#608b4e;"> Fast, precise, requires you to know exactly what you want.
</span><span style="color:#608b4e;"> """
</span><span> </span><span style="color:#569cd6;">return </span><span>EpiReport.objects.filter(
</span><span> region=region,
</span><span> disease=disease,
</span><span> year=year,
</span><span> severity=</span><span style="color:#d69d85;">"high"</span><span>,
</span><span> ).order_by(</span><span style="color:#d69d85;">"-week"</span><span>)
</span><span>
</span><span>
</span><span style="color:#608b4e;"># ── Approach 2: Semantic Vector Query ───────────────────────────────────────
</span><span>
</span><span style="color:#569cd6;">def </span><span>find_similar_reports(query: str, top_k: int = </span><span style="color:#b5cea8;">5</span><span>):
</span><span> </span><span style="color:#608b4e;">"""
</span><span style="color:#608b4e;"> Vector similarity search.
</span><span style="color:#608b4e;"> The query is natural language. No schema knowledge required.
</span><span style="color:#608b4e;"> Returns the most semantically similar reports.
</span><span style="color:#608b4e;"> """
</span><span> model = get_model()
</span><span> query_vector = model.encode(query).tolist()
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span>(
</span><span> EpiReport.objects
</span><span> .annotate(distance=CosineDistance(</span><span style="color:#d69d85;">"embedding"</span><span>, query_vector))
</span><span> .order_by(</span><span style="color:#d69d85;">"distance"</span><span>)[:top_k]
</span><span> )
</span><span>
</span><span>
</span><span style="color:#569cd6;">def </span><span>find_similar_to_report(report: EpiReport, top_k: int = </span><span style="color:#b5cea8;">5</span><span>):
</span><span> </span><span style="color:#608b4e;">"""
</span><span style="color:#608b4e;"> Given an existing report, find others that are semantically similar.
</span><span style="color:#608b4e;"> Useful for "related reports" features or duplicate detection.
</span><span style="color:#608b4e;"> """
</span><span> </span><span style="color:#569cd6;">return </span><span>(
</span><span> EpiReport.objects
</span><span> .exclude(pk=report.pk)
</span><span> .annotate(distance=CosineDistance(</span><span style="color:#d69d85;">"embedding"</span><span>, report.embedding))
</span><span> .order_by(</span><span style="color:#d69d85;">"distance"</span><span>)[:top_k]
</span><span> )
</span></code></pre>
<h3 id="wiring-it-into-views">Wiring it into views</h3>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># surveillance/views.py
</span><span style="color:#9b9b9b;">from </span><span>django.http </span><span style="color:#9b9b9b;">import </span><span>JsonResponse
</span><span style="color:#9b9b9b;">from .</span><span>models </span><span style="color:#9b9b9b;">import </span><span>EpiReport
</span><span style="color:#9b9b9b;">from .</span><span>queries </span><span style="color:#9b9b9b;">import </span><span>get_high_severity_reports, find_similar_reports
</span><span>
</span><span style="color:#569cd6;">def </span><span>structured_search(request):
</span><span> </span><span style="color:#608b4e;">"""Traditional form-based search."""
</span><span> reports = get_high_severity_reports(
</span><span> region=request.GET.get(</span><span style="color:#d69d85;">"region"</span><span>, </span><span style="color:#d69d85;">""</span><span>),
</span><span> disease=request.GET.get(</span><span style="color:#d69d85;">"disease"</span><span>, </span><span style="color:#d69d85;">""</span><span>),
</span><span> year=int(request.GET.get(</span><span style="color:#d69d85;">"year"</span><span>, </span><span style="color:#b5cea8;">2024</span><span>)),
</span><span> )
</span><span> </span><span style="color:#569cd6;">return </span><span>JsonResponse({
</span><span> </span><span style="color:#d69d85;">"approach"</span><span>: </span><span style="color:#d69d85;">"structured"</span><span>,
</span><span> </span><span style="color:#d69d85;">"results"</span><span>: list(reports.values(</span><span style="color:#d69d85;">"title"</span><span>, </span><span style="color:#d69d85;">"region"</span><span>, </span><span style="color:#d69d85;">"disease"</span><span>, </span><span style="color:#d69d85;">"severity"</span><span>))
</span><span> })
</span><span>
</span><span style="color:#569cd6;">def </span><span>semantic_search(request):
</span><span> </span><span style="color:#608b4e;">"""Free-text semantic search."""
</span><span> query = request.GET.get(</span><span style="color:#d69d85;">"q"</span><span>, </span><span style="color:#d69d85;">""</span><span>)
</span><span> reports = find_similar_reports(query, top_k=</span><span style="color:#b5cea8;">5</span><span>)
</span><span> </span><span style="color:#569cd6;">return </span><span>JsonResponse({
</span><span> </span><span style="color:#d69d85;">"approach"</span><span>: </span><span style="color:#d69d85;">"vector"</span><span>,
</span><span> </span><span style="color:#d69d85;">"query"</span><span>: query,
</span><span> </span><span style="color:#d69d85;">"results"</span><span>: [
</span><span> {
</span><span> </span><span style="color:#d69d85;">"title"</span><span>: r.title,
</span><span> </span><span style="color:#d69d85;">"region"</span><span>: r.region,
</span><span> </span><span style="color:#d69d85;">"disease"</span><span>: r.disease,
</span><span> </span><span style="color:#d69d85;">"similarity"</span><span>: round(</span><span style="color:#b5cea8;">1 </span><span>- r.distance, </span><span style="color:#b5cea8;">3</span><span>), </span><span style="color:#608b4e;"># convert distance to similarity
</span><span> }
</span><span> </span><span style="color:#569cd6;">for </span><span>r </span><span style="color:#569cd6;">in </span><span>reports
</span><span> ]
</span><span> })
</span></code></pre>
<p>Now try these two requests:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>GET /search/structured/?region=Dakar&disease=cholera&year=2024
</span><span>GET /search/semantic/?q=waterborne disease outbreaks in coastal areas
</span></code></pre>
<p>The structured search finds exactly what you specified. The semantic search finds what you <em>meant</em>, even if those exact words don't appear anywhere in your data. The world is beautiful.</p>
<hr />
<h2 id="when-to-use">When to use which</h2>
<p>On top of my tiny experience, and as a software engineer who has never used this technique on <em>on production</em>, I would say the right answer for most systems is both. And pgvector makes that even more practical without adding infrastructure complexity.</p>
<p><strong>Lean on structured queries</strong> when you need exactness: filtering by date range, counting cases, producing regulatory reports, joining across tables. SQL is unbeatable here.</p>
<p><strong>Lean on vector queries</strong> when you need semantic understanding: search boxes, recommendation engines, duplicate detection, anomaly flagging, clustering, or any time a user's intent can't be reduced to a dropdown value.</p>
<p>Even more interesting, <strong>hybrid search</strong>: use structured filters to narrow the candidate set, then rank the survivors by vector similarity:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>hybrid_search(region: str, query: str, top_k: int = </span><span style="color:#b5cea8;">5</span><span>):
</span><span> </span><span style="color:#608b4e;">"""
</span><span style="color:#608b4e;"> Filter structurally, then rank semantically.
</span><span style="color:#608b4e;"> This is typically the most useful approach in production.
</span><span style="color:#608b4e;"> """
</span><span> model = get_model()
</span><span> query_vector = model.encode(query).tolist()
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span>(
</span><span> EpiReport.objects
</span><span> .filter(region=region) </span><span style="color:#608b4e;"># hard filter: only this region
</span><span> .annotate(distance=CosineDistance(</span><span style="color:#d69d85;">"embedding"</span><span>, query_vector))
</span><span> .order_by(</span><span style="color:#d69d85;">"distance"</span><span>)[:top_k]
</span><span> )
</span></code></pre>
<hr />
<h2 id="going-further">Going further</h2>
<p>Once you've internalized the two-lens model, here's where to take it next:</p>
<h3 id="1-fine-tune-your-own-embedding-model-on-domain-data">1. Fine-tune your own embedding model on domain data</h3>
<p>The <code>all-MiniLM-L6-v2</code> <a href="https://www.sbert.net/">model</a> was trained on general English text. It has no idea that "paludisme" and "malaria" are the same thing, or that "cas suspects" and "suspected cases" are equivalent in my context. Training a domain-specific embedding model on my own epidemiological corpus — even with a small dataset using contrastive learning — would make similarity search dramatically more meaningful.</p>
<h3 id="2-retrieval-augmented-generation-rag-on-top-of-our-vector-store">2. Retrieval-augmented generation (RAG) on top of our vector store</h3>
<p>Once we have a vector store of reports, we're one step away from a system that can <em>answer questions</em> about your data in natural language. A user asks: <em>"What were the major respiratory disease trends in Guinea-Bissau in 2023?"</em> — our system retrieves the most relevant reports by vector similarity, then feeds them as context to an <a href="https://en.wikipedia.org/wiki/Large_language_model">LLM</a> that synthesizes a coherent answer.</p>
<h3 id="3-vectors-for-anomaly-detection-in-surveillance-data">3. Vectors for anomaly detection in surveillance data</h3>
<p>Every week a new batch of case reports comes in. <a href="https://nskm.xyz/posts/syndromic-surveillance/#engineers">We</a> can embed each report and compare it to the historical distribution of vectors for that region and disease. A report that lands far from the cluster is a statistical anomaly, potentially a real <a href="https://www.who.int/activities/epidemic-and-pandemic-monitoring">outbreak signal</a>. This turns our vector store into a passive early-warning system without writing any explicit outbreak-detection rules.</p>
<h3 id="4-multimodal-embeddings">4. Multimodal embeddings</h3>
<p>Text isn't the only thing we can embed. We can embed laboratory results, genomic sequences, geographic coordinates combined with case counts, even scanned paper forms via vision models. The moment we have multiple modalities in the same vector space, we can do cross-modal search: <em>find reports whose geographic spread pattern resembles this map image.</em></p>
<p>placeholder image for brain explosion</p>
<h3 id="5-the-theory-underneath-why-vector-spaces-work">5. The theory underneath: why vector spaces work</h3>
<hr />
<h2 id="broader-landscape">Representation models: other ways to represent data</h2>
<p>Structured attributes and vectors are two lenses. And there are other ways to represent data:</p>
<table style="border-collapse: collapse; width: 100%;">
<tr style="border: 1px solid #ddd;">
<th style="border: 1px solid #ddd; padding: 12px; text-align: left;">Representation</th>
<th style="border: 1px solid #ddd; padding: 12px; text-align: left;">Best question it answers</th>
<th style="border: 1px solid #ddd; padding: 12px; text-align: left;">Tools</th>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Class + attributes</td>
<td style="border: 1px solid #ddd; padding: 12px;">What does this thing have?</td>
<td style="border: 1px solid #ddd; padding: 12px;">Django ORM, PostgreSQL</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Vector</td>
<td style="border: 1px solid #ddd; padding: 12px;">What is this thing like?</td>
<td style="border: 1px solid #ddd; padding: 12px;">pgvector, sentence-transformers</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Graph</td>
<td style="border: 1px solid #ddd; padding: 12px;">How does this thing connect?</td>
<td style="border: 1px solid #ddd; padding: 12px;">Neo4j, networkx</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Event stream</td>
<td style="border: 1px solid #ddd; padding: 12px;">How did this thing become what it is?</td>
<td style="border: 1px solid #ddd; padding: 12px;">Kafka, EventStoreDB</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Logic / facts</td>
<td style="border: 1px solid #ddd; padding: 12px;">What can we conclude about this thing?</td>
<td style="border: 1px solid #ddd; padding: 12px;">Prolog, Datalog, RDF/OWL</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Probability</td>
<td style="border: 1px solid #ddd; padding: 12px;">How certain are we about this thing?</td>
<td style="border: 1px solid #ddd; padding: 12px;">PyMC, Stan</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Spatial</td>
<td style="border: 1px solid #ddd; padding: 12px;">Where is this thing, relative to what?</td>
<td style="border: 1px solid #ddd; padding: 12px;">PostGIS, GeoDjango</td>
</tr>
<tr style="border: 1px solid #ddd;">
<td style="border: 1px solid #ddd; padding: 12px;">Tensor</td>
<td style="border: 1px solid #ddd; padding: 12px;">How does this vary across multiple dimensions?</td>
<td style="border: 1px solid #ddd; padding: 12px;">NumPy, PyTorch, Xarray</td>
</tr>
</table>
<hr />
<h2 id="how-to-practice">How to practice</h2>
<p>You don't need production data to start experimenting with vectors and embeddings:</p>
<p><strong>Local development:</strong></p>
<ul>
<li><strong><a href="https://www.trychroma.com/">Chroma</a></strong>: Lightweight vector database that runs locally or in Docker.</li>
<li><strong><a href="https://hub.docker.com/r/pgvector/pgvector">pgvector in Docker</a></strong>: A local PostgreSQL with pgvector extension in seconds.</li>
<li><strong><a href="https://www.sbert.net/docs/pretrained_models.html">Sentence Transformers Models</a></strong>: Experiment with domain-specific models beyond all-MiniLM-L6-v2.</li>
</ul>
<p><strong>Learning resources:</strong></p>
<ul>
<li><strong><a href="https://huggingface.co/models">Hugging Face Hub</a></strong>: 100k+ pre-trained models. Read papers, compare benchmark results.</li>
<li><strong><a href="https://huggingface.co/learn">Embedding Models Course</a></strong>: Free, practical introduction to embeddings.</li>
<li><strong><a href="https://github.com/pgvector/pgvector">pgvector Documentation</a></strong>: All indexing strategies and similarity operators explained.</li>
</ul>
<hr />
<h2 id="more-on-this-topic">More on this topic</h2>
<p>As a Django engineer, my mental model of "data as class with attributes" is solid and won't stop serving <a href="https://nskm.xyz/images/Nsukami_Patrick.pdf">me</a>. But it's worth internalizing that it's one of at least two valid representations. I should definitely start to think of my data as having a <em>position</em> in semantic space, a whole class of problems (search, similarity, anomaly, clustering) will become natural rather than bespoke.</p>
<p>Here we are, we wanted a banana but we received a gorilla holding the banana and the entire jungle. Here are some links to help you <em>(and me, let's be honest)</em> go further and tame the beast:</p>
<ul>
<li><strong><a href="https://dev.to/dev_kiran/embeddings-the-hidden-power-behind-ai-search-21eb">The hidden power behind AI & search</a></strong></li>
<li><strong><a href="https://www.sbert.net/docs/training/overview.html">Fine-tuning sentence transformers</a></strong></li>
<li><strong><a href="https://vickiboykis.com/what_are_embeddings/">Vector database comparison</a></strong></li>
<li><strong><a href="https://lilianweng.github.io/posts/2021-05-31-contrastive/">Contrastive learning explained</a></strong></li>
<li><strong><a href="https://arxiv.org/abs/2005.11401">Retrieval-Augmented Generation (RAG)</a></strong></li>
<li><strong><a href="https://github.com/pgvector/pgvector#performance">pgvector performance tuning</a></strong></li>
<li><strong><a href="https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667.html">On the foolishness of "natural language programming"</a></strong></li>
</ul>
<p>If you found this article useful, <em>please like, share and subscribe</em>.</p>
<hr />
<h2 id="while-i-still-have-your-attention">While I still have your attention</h2>
<p>The Norwegian Consumer Council is an independent, governmentally funded organisation that advocates for consumer’s rights. It should be easy for consumers to make sustainable choices every day. Consumers have the right to be protected against exploitation – both financially and digitally. To ensure this, we work to provide easy access to information, enforceable rights, and sufficient redress options when something goes wrong.</p>
<p>Digital products and services keep getting worse. In the new report Breaking Free: Pathways to a fair technological future, the Norwegian Consumer Council has delved into <a href="https://www.youtube.com/watch?v=T4Upf_B9RLQ">enshittification</a> and how to resist it. The report shows how this phenomenon affects both consumers and society at large, but that it is possible to <a href="https://www.forbrukerradet.no/breakingfree">turn the tide.</a></p>
<p><em>Geez</em>, I would have loved to see African leaders do the same....</p>
AWS (Recap)2026-02-12T00:00:00+00:002026-02-12T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-recap/<img style="display: block; margin: 0 auto; width: 300px" src="/images/proud_of_you.gif" alt="Proud of myself" title="Proud of myself"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers:</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece. If you catch any errors, please reach out — I'd be grateful for the feedback and happy to make updates!</p>
</li>
</ol>
<br>
<br>
<h2 id="hook">Hook</h2>
<p>Structural truth about intensive training formats: you can cover a lot of ground in a week, but covering ground is not the same as owning it. There is a fundamental difference between <em>understanding</em> the concepts and <em>having debugged a deployment issue at 11pm before a production deployment</em>. The gap between "I understand this" and "I can do this under pressure" is filled only by practice — and eight days of excellent training doesn't close it.</p>
<p>10 days of training, a lot of diagrams drawn on whiteboards, 5 instructors who clearly knew their material. I would like to thank them very much; I have learnt a great deal thanks to them.</p>
<p>With this article, I would like to take a step back to look at what we built, what was, <em>imho</em>, missing and what is the next step.... <a href="https://www.youtube.com/watch?v=75wmW7xjyog">Music</a> please!</p>
<br>
<br>
<h2 id="toc">ToC</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-recap/#the-platform-we-built">The platform we built</a></li>
<li><a href="https://nskm.xyz/posts/aws-recap/#why-in-this-order">Why in this order</a></li>
<li><a href="https://nskm.xyz/posts/aws-recap/#day-7-the-piece-thats-missing">Ansible: the piece that's missing</a></li>
<li><a href="https://nskm.xyz/posts/aws-recap/#what-we-didnt-cover">What we didn't cover</a></li>
<li><a href="https://nskm.xyz/posts/aws-recap/#where-to-go-from-here">Where to go from here</a></li>
<li><a href="https://nskm.xyz/posts/aws-recap/#conclusion">Conclusion</a></li>
</ol>
<p><br><br></p>
<h2 id="the-platform-we-built">The platform we built</h2>
<p>This 10 days training series actually produced a concrete thing: how to build Django REST APIs handling patient analyses and clinical consultation data, running on a K8s cluster on AWS, secured at every layer, deployable in any environment in minutes, and fully observable.</p>
<p>Here is what that means in practice:</p>
<ul>
<li>A <strong>private VPC</strong> in a region with public subnets for load balancers and private subnets for everything else</li>
<li>An <strong>EKS cluster</strong> provisioned with managed node groups, with etcd encrypted at rest using a KMS key</li>
<li><strong>IAM roles</strong> for every component that needs AWS access, <strong>IRSA or EKS Pod Identity</strong> for pods</li>
<li><strong>RBAC</strong> scoped to namespaces: the genomics team can only touch the genomics namespace, the clinical trials team theirs</li>
<li><strong>Network policies</strong> enforcing zero-trust between pods: no pod talks to another unless explicitly allowed</li>
<li><strong>Secrets</strong> stored in AWS Secrets Manager, synced into Kubernetes at runtime by the External Secrets Operator</li>
<li>The entire infrastructure <strong>provisioned by Terraform</strong> — reproducible in any environment</li>
<li>The Django Rest APIs <strong>packaged as a Helm charts</strong></li>
<li>The charts stored in <strong>ECR alongside the Docker images</strong>, accessible to any teammate with the right IAM permissions</li>
<li>Platform <strong>instrumented with OpenTelemetry</strong>: metrics sent to <strong>Prometheus</strong>, logs to <strong>Loki</strong>, traces to <strong>Jaeger</strong>, all visualised in <strong>Grafana</strong></li>
<li><strong>SLOs</strong> defined for the Rest APIs, with Alertmanager firing when the error budget burns too fast</li>
</ul>
<p><br><br></p>
<h2 id="why-in-this-order">What we built, why in this order</h2>
<p>Layers on layers on layers. Each layer depended on the one before it.</p>
<p><strong><a href="https://nskm.xyz/posts/aws-1/">Foundations</a></strong> — VPC, compute, storage, regions. Before anything else, you need to understand the environment your workloads will run in. Network topology is a first-class architectural decision.</p>
<p><strong><a href="https://nskm.xyz/posts/aws-2-3/">Identity and access</a></strong> — IAM, OIDC, KMS, CloudTrail. IAM roles and encryption are established <em>before</em> the cluster exists, because the cluster will inherit them from day one. It's difficult to retroactively secure a system that was built without security in mind.</p>
<p><strong><a href="https://nskm.xyz/posts/aws-4/">Orchestration</a></strong> — Kubernetes, EKS, pods, deployments, services. To run workloads at scale, self-heal them, and expose them, without managing individual servers.</p>
<p><strong><a href="https://nskm.xyz/posts/aws-5/">Workload security</a></strong> — RBAC, network policies, pod security, secrets management. You have a running cluster. Now lock down what runs inside it. The defaults are permissive by design; the hardening is your responsibility.</p>
<p><strong><a href="https://nskm.xyz/posts/aws-6/">Automation</a></strong> — Terraform, Helm. You have built & secured a cluster you understand. Now make it reproducible. The goal is to never click through the console again — for infrastructure or for deployments.</p>
<p><strong><a href="https://nskm.xyz/posts/aws-8/">Observability</a></strong> — Prometheus, Loki, Jaeger, OpenTelemetry, SLOs. The platform runs, is secured, and is automated. Now make it visible. You cannot maintain what you cannot see.</p>
<p>Mental model: <strong>build → orchestrate → secure → automate → observe</strong>. Each verb assumes the previous one is done. You cannot automate what you have not secured. And you do not observe what you have not deployed. <em>Phew!</em></p>
<p><br><br></p>
<h2 id="day-7-the-piece-thats-missing">Ansible: the piece that's missing</h2>
<p>Somewhere during the training, we covered <strong>Ansible</strong>. I chose to not talk about it as part of this series.</p>
<p>Yes, Ansible is useful and complementary to Terraform, Helm & Kubernetes, but not strictly essential. If you are using fully managed Kubernetes services (like AWS EKS or GKE) where the cloud provider manages the underlying node configuration, the need for Ansible is reduced.</p>
<p>On Kubernetes, the configuration management layer is largely handled by Helm values, ConfigMaps, and Secrets. Ansible's strengths shine on EC2 instances, bastion hosts, and hybrid environments — which were not the primary focus of this training.</p>
<p>In all the cases, I'm not saying Ansible is worthless, it is absolutely worth learning. The <a href="https://docs.ansible.com/">official documentation</a> and the <a href="https://www.ansiblefordevops.com/">Ansible for DevOps book</a> by Jeff Geerling are the right starting points.</p>
<p><br><br></p>
<h2 id="what-we-didnt-cover">What we didn't cover</h2>
<p>While I agree the instructors gave the best they could, while I agree we cannot cover everything in 10 days, the following topics are the ones I wish were addressed:</p>
<p><strong>CI/CD pipelines</strong> — how does a git push become a Docker image in ECR and a Helm upgrade on the cluster? I know 1 or 2 things about this topic but I wish the instructors had said a few words about that.</p>
<p><strong>GitOps</strong> — <a href="https://argoproj.github.io/cd/">ArgoCD</a> and <a href="https://fluxcd.io/">Flux</a> take Helm one step further: Git becomes the single source of truth, and the cluster continuously reconciles itself against what's in the repository. This is how most mature platform teams deploy today.</p>
<p><strong>Service mesh</strong> — <a href="https://istio.io/">Istio</a>, <a href="https://linkerd.io/">Linkerd</a>, and <a href="https://cilium.io/">Cilium</a> add mTLS between services, fine-grained traffic management, and advanced observability at the network layer. Once your platform has dozens of services, a service mesh stops being optional.</p>
<p><strong>Backup and restore</strong> — <a href="https://velero.io/">Velero</a> backs up Kubernetes resources and persistent volumes. What happens when someone deletes the wrong namespace in production?</p>
<p><strong>Multi-region and disaster recovery</strong> — what happens when <code>eu-west-3</code> has an incident? For a platform handling patient data, this is a compliance question as much as a technical one.</p>
<p><strong>Image scanning in CI</strong> — <a href="https://trivy.dev/">Trivy</a> and <a href="https://snyk.io/">Snyk</a> scan Docker images for known vulnerabilities before they ever reach the cluster. Security in the build pipeline, not just at runtime.</p>
<p><br><br></p>
<h2 id="where-to-go-from-here">Where to go from here</h2>
<p>Then, explore <strong>ArgoCD</strong>. Point it at an Helm chart repository in ECR, and let it manage deployments declaratively. That's the missing link between Day 6 and a production-grade deployment workflow.</p>
<p>Finally, set up the observability stack from <a href="https://nskm.xyz/posts/aws-8/">Day 8</a> and use it to actually find out why/when something broke. Today, I'll be honest, if something break, The only thing I have to help me sort this out are the <a href="https://i.programmerhumor.io/2025/11/6d58f6fc8886c2da6ee0215f5a2e00207595c36ef43031d9e9ec33741120dcc3.jpg">logs</a>.</p>
<img style="display: block; margin: 0 auto; width: 300px" src="/images/writing.gif" alt="Planning the next step" title="Planning the next step"/>
<p><br><br></p>
<h2 id="conclusion">Conclusion</h2>
<p>Ten days of excellent content, delivered by people who clearly knew what they were talking about, thanks a million to them. I know the vocabulary, I still need to pratice, a lot, to own the skills. Writing these articles was the first step in that direction. Now <em>I need a credit card.</em></p>
<br>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>You know the drill.... see ya o/</p>
<p><strong>CI/CD and GitOps:</strong></p>
<ul>
<li><a href="https://docs.github.com/en/actions">GitHub Actions documentation</a></li>
<li><a href="https://argo-cd.readthedocs.io/">ArgoCD documentation</a></li>
<li><a href="https://fluxcd.io/docs/">Flux documentation</a></li>
<li><a href="https://www.youtube.com/watch?v=MeU5_k9ssrs">GitOps with ArgoCD — tutorial</a></li>
</ul>
<p><strong>Service mesh:</strong></p>
<ul>
<li><a href="https://istio.io/latest/docs/">Istio documentation</a></li>
<li><a href="https://linkerd.io/docs/">Linkerd documentation</a></li>
<li><a href="https://docs.cilium.io/">Cilium documentation</a></li>
</ul>
<p><strong>Cost and autoscaling:</strong></p>
<ul>
<li><a href="https://www.kubecost.com/">Kubecost</a></li>
<li><a href="https://karpenter.sh/docs/">Karpenter documentation</a></li>
</ul>
<p><strong>Backup and resilience:</strong></p>
<ul>
<li><a href="https://velero.io/docs/">Velero documentation</a></li>
</ul>
<p><strong>Going deeper on what was covered:</strong></p>
<ul>
<li><a href="https://aws.github.io/aws-eks-best-practices/">AWS EKS best practices guide</a> — the authoritative reference for everything in Days 4 and 5</li>
<li><a href="https://sre.google/sre-book/table-of-contents/">Google SRE book</a> — the canonical text behind the SLI/SLO/SLA framework from Day 8</li>
<li><a href="https://www.ansiblefordevops.com/">Ansible for DevOps</a> — Jeff Geerling's book, the right starting point for Day 7</li>
<li><a href="https://terraformbook.com/">The Terraform Book</a> — for going deeper on Day 6</li>
</ul>
<br>
<br>
<img style="display: block; margin: 0 auto; width: 500px" src="/images/modern_infrastructure.webp" alt="Modern infrastructure" title="Modern infrastructure"/>
AWS (Day 8)2026-02-11T00:00:00+00:002026-02-11T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-8/ <img style="display: block; margin: 0 auto; width: 400px" src="/images/observability2.gif" alt="Me, observing" title="Me observing"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>This article is <strong>educational content</strong>. The examples are intentionally simplified for clarity. All tools featured here are free and open-source software — they run identically on EKS, GKE, bare metal, or your laptop.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece. If you catch any errors, please reach out — I'd be grateful for the feedback and happy to make updates!</p>
</li>
<li>
<p>A special thank you to <a href="https://www.linkedin.com/in/mameastougassama/">Mame Astou</a> aka "Madame La Directrice" for taking the time to review this article.
<br></p>
</li>
</ol>
<h2 id="hook">Hook</h2>
<p><em>The genomics API is running. <a href="https://nskm.xyz/posts/aws-6/">Terraform provisioned the cluster</a>, <a href="https://nskm.xyz/posts/aws-6/#helm-kubernetes-package-manager">Helm deployed the application</a>. From the outside, everything looks fine.</em></p>
<p><em>Then, a researcher reports that their variant analysis job submitted two hours ago still hasn't returned results. You check Kubernetes: the pods are running. You check the load balancer: it's healthy. The deployment logs show no errors. But <em>something</em> is wrong, and you have absolutely no idea where to start looking.</em></p>
<p>That's the moment you realise your system is <strong>blind</strong>. You built the infrastructure. You secured it. You automated its deployment. And then you flew it into production with no instruments.</p>
<p>This article is about fixing that. Not by adding monitoring as an afterthought, but by building the kind of visibility that lets you answer three questions before your users even notice a problem: <strong>what is happening, why is it happening, and where exactly is it happening?</strong> No worries, I'm not there yet. Lord knows how much 4s misses observability & monitoring :\</p>
<br>
<br>
<h2 id="toc">ToC</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-8/#observability-vs-monitoring">Observability vs monitoring</a></li>
<li><a href="https://nskm.xyz/posts/aws-8/#the-four-pillars">The four pillars</a></li>
<li><a href="https://nskm.xyz/posts/aws-8/#the-tooling-landscape">The tooling landscape</a></li>
<li><a href="https://nskm.xyz/posts/aws-8/#opentelemetry-the-collection-standard">OpenTelemetry: the collection standard</a></li>
<li><a href="https://nskm.xyz/posts/aws-8/#sli-slo-sla-from-data-to-accountability">SLI, SLO, SLA: from data to accountability</a></li>
<li><a href="https://nskm.xyz/posts/aws-8/#conclusion">Conclusion</a></li>
<li><a href="https://nskm.xyz/posts/aws-8/#more-on-this-topic">More on this topic</a></li>
</ol>
<p><br><br></p>
<h2 id="observability-vs-monitoring">Observability vs monitoring</h2>
<p>These two words are often used interchangeably. They shouldn't be.</p>
<p><strong>Monitoring</strong> is about watching <em>known</em> failure modes. You define thresholds — CPU above 80%, error rate above 1% — and you alert when they're crossed. It answers the question: <em>"is this thing I already know about happening?"</em> Monitoring is reactive: you first have to know what can go wrong before you can monitor for it.</p>
<p><strong>Observability</strong> is about understanding <em>unknown</em> failure modes from the outside. A system is observable if you can ask arbitrary questions about its internal state — without having predicted those questions in advance. It answers: <em>"why is this behaving the way it is?"</em></p>
<p>In practice, you need both. Monitoring te dit "quelque chose est cassé". Observability te permet de répondre à "pourquoi est-ce cassé, exactement, pour qui, depuis quand, et dans quel contexte" — même pour des pannes que tu n'avais pas anticipées.</p>
<p>In a K8s cluster, failures are often not the ones you predicted. A genomic analysis pipeline stalls not because of high CPU, but because a database connection pool is exhausted by a single slow query in one namespace — and your monitoring threshold was on the wrong metric entirely. Observability gives you the tools to discover that. Monitoring alone doesn't.</p>
<img style="display: block; margin: 0 auto; width: 600px" src="/images/observability_diagram.svg" alt="observability" title="observability"/>
<p><br><br></p>
<img style="display: block; margin: 0 auto; width: 600px" src="/images/monitoring_diagram.svg" alt="monitoring" title="monitoring"/>
<p><br><br></p>
<h2 id="the-four-pillars">The four pillars</h2>
<p>Cloud-native observability is built on four pillars. Together, they give you a complete picture of a running system.</p>
<h3 id="logs-what-happened">Logs — what happened</h3>
<p><a href="https://opentelemetry.io/docs/concepts/signals/logs/">Logs</a> are timestamped records of discrete events: a request came in, a query failed, a job completed. They are the most familiar signal and the easiest to produce — every Django app already writes logs.</p>
<p>On Kubernetes, logs have a structural challenge: pods are ephemeral. When a pod crashes and restarts, its logs disappear. You need a <strong>log aggregation layer</strong> that collects logs from all pods, across all nodes, and stores them centrally before they vanish.</p>
<p><em>Answers: "What exactly happened at 14:32:07 in the genomics namespace?"</em></p>
<h3 id="metrics-how-much">Metrics — how much</h3>
<p><a href="https://opentelemetry.io/docs/concepts/signals/metrics/">Metrics</a> are numeric measurements sampled over time: request rate, error rate, latency percentiles, CPU usage, memory consumption, active database connections. They are cheap to store, fast to query, and ideal for alerting and dashboards.</p>
<p>Metrics don't tell you <em>why</em> something is slow — but they tell you <em>that</em> it is slow and <em>when</em> it started.</p>
<p><em>Answers: "How many analysis requests failed in the last hour, and is that number growing?"</em></p>
<h3 id="traces-the-full-journey">Traces — the full journey</h3>
<p>A <a href="https://opentelemetry.io/docs/concepts/signals/traces/">trace</a> follows a single request through every service it touches. In a microservices or multi-component architecture, a single API call might touch a Django view, a Celery task, a PostgreSQL query, and an S3 upload. A trace connects all of these into a single timeline, with durations for each step.</p>
<p>Traces are the pillar that answers the question logs and metrics can't: <em>"the request was slow — but which part of it was slow?"</em></p>
<p><em>Answers: "The variant analysis request took 4.2 seconds — 3.8 seconds of which was a single database query in the pipeline."</em></p>
<h3 id="kubernetes-events-what-the-cluster-decided">Kubernetes events — what the cluster decided</h3>
<p>Kubernetes <a href="https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/">emits events</a> for everything it does: pod scheduled, pod evicted, image pull failed, node pressure detected, HPA scaled up. These events are not logs (they come from the control plane, not the application) and not metrics (they are discrete, not continuous). They are a fourth pillar that explains <em>cluster-level</em> behaviour.</p>
<p><em>Answers: "The pod restarted three times because the node ran out of memory — not because the application crashed."</em></p>
<p><br><br></p>
<h2 id="the-tooling-landscape">The tooling landscape</h2>
<p>Two paths exist for implementing this stack on Kubernetes.</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Signal</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">AWS-native</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Open source</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Logs</strong></td>
<td style="border: 1px solid #333; padding: 10px;">CloudWatch Logs</td>
<td style="border: 1px solid #333; padding: 10px;">Loki + Promtail</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Metrics</strong></td>
<td style="border: 1px solid #333; padding: 10px;">CloudWatch Metrics</td>
<td style="border: 1px solid #333; padding: 10px;">Prometheus + node-exporter</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Traces</strong></td>
<td style="border: 1px solid #333; padding: 10px;">AWS X-Ray</td>
<td style="border: 1px solid #333; padding: 10px;">Jaeger</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Visualisation</strong></td>
<td style="border: 1px solid #333; padding: 10px;">CloudWatch Dashboards</td>
<td style="border: 1px solid #333; padding: 10px;">Grafana</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Instrumentation</strong></td>
<td style="border: 1px solid #333; padding: 10px;">AWS Distro for OpenTelemetry</td>
<td style="border: 1px solid #333; padding: 10px;">OpenTelemetry</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Alerting</strong></td>
<td style="border: 1px solid #333; padding: 10px;">CloudWatch Alarms</td>
<td style="border: 1px solid #333; padding: 10px;">Prometheus Alertmanager</td>
</tr>
</tbody>
</table>
<p>We are going with the <strong>open-source stack</strong>. These tools run identically on EKS, GKE, bare metal, or your laptop. They have no vendor lock-in, no per-metric pricing, and a massive community behind each of them. If the genomics platform ever moves off AWS, the observability layer moves with it unchanged.</p>
<h3 id="how-the-pieces-fit-together">How the pieces fit together</h3>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>Django app
</span><span> └─ OTel SDK (auto-instrumentation)
</span><span> └─ OTel Collector
</span><span> ├─ metrics ──→ Prometheus ──→ Grafana
</span><span> ├─ logs ─────→ Loki ─────────→ Grafana
</span><span> └─ traces ───→ Jaeger ────────→ Grafana
</span><span> Alertmanager ──→ PagerDuty / Slack
</span></code></pre>
<p>A single <strong>OTel Collector</strong> acts as the central routing layer — your application sends all three signal types to one endpoint, and the collector fans them out to the right backends. Grafana sits on top of all three, giving you a unified view. This architecture means your application code only knows about one destination: the collector.</p>
<h3 id="deploying-the-stack-with-helm">Deploying the stack with Helm</h3>
<p><a href="https://nskm.xyz/posts/aws-6/">From Day 6</a>, you already know how to use Helm. The entire observability stack is available as Helm charts:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
</span><span>helm repo add grafana https://grafana.github.io/helm-charts
</span><span>helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
</span><span>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
</span><span>helm repo update
</span><span>
</span><span style="color:#608b4e;"># Prometheus + Grafana + Alertmanager — all in one chart
</span><span>helm install kube-prometheus-stack \
</span><span> prometheus-community/kube-prometheus-stack \
</span><span> --namespace monitoring \
</span><span> --create-namespace
</span><span>
</span><span style="color:#608b4e;"># Loki + Promtail (log collection from all pods)
</span><span>helm install loki grafana/loki-stack \
</span><span> --namespace monitoring \
</span><span> --set promtail.enabled=true
</span><span>
</span><span style="color:#608b4e;"># Jaeger (distributed tracing backend)
</span><span>helm install jaeger jaegertracing/jaeger \
</span><span> --namespace monitoring
</span><span>
</span><span style="color:#608b4e;"># OpenTelemetry Collector (the routing layer)
</span><span>helm install otel-collector \
</span><span> open-telemetry/opentelemetry-collector \
</span><span> --namespace monitoring \
</span><span> --values otel-collector-values.yaml
</span></code></pre>
<p><br><br></p>
<h2 id="opentelemetry-the-collection-standard">OpenTelemetry: the collection standard</h2>
<p><a href="https://opentelemetry.io/">OpenTelemetry</a> (OTel) is a CNCF project that provides a single, vendor-neutral standard for producing and collecting observability data. Before OTel, you needed separate SDKs for Prometheus, Jaeger, and your logging library — each with different APIs and configuration. OTel unifies them: one SDK, one wire protocol, one collector.</p>
<p>It has two parts: the <strong>SDK</strong> (what you add to your application to produce telemetry) and the <strong>Collector</strong> (the agent that receives, processes, and exports it).</p>
<h3 id="the-otel-collector-central-routing">The OTel Collector: central routing</h3>
<p>The collector is configured as a pipeline: receivers → processors → exporters. Here is the configuration for our stack (<code>otel-collector-values.yaml</code>):</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">config</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">receivers</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">otlp</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">protocols</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">grpc</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">endpoint</span><span>: </span><span style="background-color:#282828;color:#d69d85;">0.0.0.0:4317</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">http</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">endpoint</span><span>: </span><span style="background-color:#282828;color:#d69d85;">0.0.0.0:4318</span><span>
</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">processors</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">batch</span><span>: {}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory_limiter</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">limit_mib</span><span>: </span><span style="color:#b5cea8;">512
</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">exporters</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">prometheus</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">endpoint</span><span>: </span><span style="color:#d69d85;">"0.0.0.0:8889"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">loki</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">endpoint</span><span>: </span><span style="background-color:#282828;color:#d69d85;">http://loki:3100/loki/api/v1/push</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">jaeger</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">endpoint</span><span>: </span><span style="background-color:#282828;color:#d69d85;">jaeger-collector:14250</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">tls</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">insecure</span><span>: </span><span style="color:#569cd6;">true
</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pipelines</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">metrics</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">receivers</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">otlp</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">processors</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">memory_limiter</span><span>, </span><span style="background-color:#282828;color:#d69d85;">batch</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">exporters</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">prometheus</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">logs</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">receivers</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">otlp</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">processors</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">memory_limiter</span><span>, </span><span style="background-color:#282828;color:#d69d85;">batch</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">exporters</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">loki</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">traces</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">receivers</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">otlp</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">processors</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">memory_limiter</span><span>, </span><span style="background-color:#282828;color:#d69d85;">batch</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">exporters</span><span>: [</span><span style="background-color:#282828;color:#d69d85;">jaeger</span><span>]
</span></code></pre>
<p>Three pipelines, three backends — your application only talks to port 4317.</p>
<h3 id="instrumenting-the-django-api">Instrumenting the Django API</h3>
<p>The genomics API needs two things: a way to produce telemetry (the OTel SDK) and a way to expose metrics for Prometheus to scrape (the <code>django-prometheus</code> package).</p>
<p><strong>Dependencies:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>pip install \
</span><span> opentelemetry-sdk \
</span><span> opentelemetry-instrumentation-django \
</span><span> opentelemetry-instrumentation-psycopg2 \
</span><span> opentelemetry-exporter-otlp-proto-grpc \
</span><span> django-prometheus
</span></code></pre>
<p><strong>Auto-instrumentation setup</strong> — the right place to call this is <code>AppConfig.ready()</code> in your app's <code>apps.py</code>, which runs exactly once on startup regardless of whether you're using <code>manage.py</code>, a WSGI server, or an ASGI server:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">from </span><span>opentelemetry </span><span style="color:#9b9b9b;">import </span><span>trace
</span><span style="color:#9b9b9b;">from </span><span>opentelemetry.sdk.trace </span><span style="color:#9b9b9b;">import </span><span>TracerProvider
</span><span style="color:#9b9b9b;">from </span><span>opentelemetry.sdk.trace.export </span><span style="color:#9b9b9b;">import </span><span>BatchSpanProcessor
</span><span style="color:#9b9b9b;">from </span><span>opentelemetry.exporter.otlp.proto.grpc.trace_exporter </span><span style="color:#9b9b9b;">import </span><span>OTLPSpanExporter
</span><span style="color:#9b9b9b;">from </span><span>opentelemetry.instrumentation.django </span><span style="color:#9b9b9b;">import </span><span>DjangoInstrumentor
</span><span style="color:#9b9b9b;">from </span><span>opentelemetry.instrumentation.psycopg2 </span><span style="color:#9b9b9b;">import </span><span>Psycopg2Instrumentor
</span><span>
</span><span style="color:#569cd6;">def </span><span>configure_tracing():
</span><span> provider = TracerProvider()
</span><span> exporter = OTLPSpanExporter(
</span><span> endpoint=</span><span style="color:#d69d85;">"http://otel-collector.monitoring:4317"</span><span>,
</span><span> insecure=</span><span style="color:#569cd6;">True
</span><span> )
</span><span> provider.add_span_processor(BatchSpanProcessor(exporter))
</span><span> trace.set_tracer_provider(provider)
</span><span>
</span><span> </span><span style="color:#608b4e;"># Auto-instrument Django HTTP requests and PostgreSQL queries
</span><span> DjangoInstrumentor().instrument()
</span><span> Psycopg2Instrumentor().instrument()
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># yourapp/apps.py
</span><span style="color:#9b9b9b;">from </span><span>django.apps </span><span style="color:#9b9b9b;">import </span><span>AppConfig
</span><span>
</span><span style="color:#569cd6;">class </span><span>YourAppConfig(</span><span style="color:#4ec9b0;">AppConfig</span><span>):
</span><span> name = </span><span style="color:#d69d85;">"yourapp"
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>ready(self):
</span><span> configure_tracing()
</span></code></pre>
<p>With this in place, every HTTP request and every database query is automatically traced — no manual instrumentation needed for the infrastructure layer.</p>
<p><strong>Custom spans for business logic</strong> — the above covers HTTP and database calls automatically. For domain-specific visibility into your own code, add manual spans:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># analysis/views.py
</span><span style="color:#9b9b9b;">from </span><span>opentelemetry </span><span style="color:#9b9b9b;">import </span><span>trace
</span><span>
</span><span>tracer = trace.get_tracer(__name__)
</span><span>
</span><span style="color:#569cd6;">def </span><span>run_variant_analysis(request, patient_id):
</span><span> </span><span style="color:#569cd6;">with </span><span>tracer.start_as_current_span(</span><span style="color:#d69d85;">"variant-analysis"</span><span>) </span><span style="color:#569cd6;">as </span><span>span:
</span><span> span.set_attribute(</span><span style="color:#d69d85;">"patient.id"</span><span>, patient_id)
</span><span> span.set_attribute(</span><span style="color:#d69d85;">"analysis.type"</span><span>, </span><span style="color:#d69d85;">"variant-calling"</span><span>)
</span><span>
</span><span> result = VariantCallingPipeline.run(patient_id)
</span><span>
</span><span> span.set_attribute(</span><span style="color:#d69d85;">"analysis.variants_found"</span><span>, result.variant_count)
</span><span> span.set_attribute(</span><span style="color:#d69d85;">"analysis.duration_ms"</span><span>, result.duration_ms)
</span><span> </span><span style="color:#569cd6;">return </span><span>JsonResponse(result.to_dict())
</span></code></pre>
<p>Now when an analysis is slow, the trace shows exactly where the time went: Django view setup, PostgreSQL query, pipeline execution, response serialisation — each as a separate span with its own duration.</p>
<h3 id="exposing-metrics-for-prometheus">Exposing metrics for Prometheus</h3>
<p>Add <code>django-prometheus</code> to <code>INSTALLED_APPS</code> and wire up the metrics endpoint:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># settings.py
</span><span>INSTALLED_APPS = [
</span><span> </span><span style="color:#569cd6;">...
</span><span> </span><span style="color:#d69d85;">'django_prometheus'</span><span>,
</span><span>]
</span><span>
</span><span>MIDDLEWARE = [
</span><span> </span><span style="color:#d69d85;">'django_prometheus.middleware.PrometheusBeforeMiddleware'</span><span>,
</span><span> </span><span style="color:#569cd6;">...
</span><span> </span><span style="color:#d69d85;">'django_prometheus.middleware.PrometheusAfterMiddleware'</span><span>,
</span><span>]
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># urls.py
</span><span style="color:#9b9b9b;">from </span><span>django.urls </span><span style="color:#9b9b9b;">import </span><span>path, include
</span><span>
</span><span>urlpatterns = [
</span><span> path(</span><span style="color:#d69d85;">''</span><span>, include(</span><span style="color:#d69d85;">'django_prometheus.urls'</span><span>)), </span><span style="color:#608b4e;"># exposes /metrics
</span><span> </span><span style="color:#569cd6;">...
</span><span>]
</span></code></pre>
<p>Then tell Prometheus to scrape it with a <code>ServiceMonitor</code> (from the <code>kube-prometheus-stack</code> CRDs):</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">monitoring.coreos.com/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ServiceMonitor</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-api</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">monitoring</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-api</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespaceSelector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchNames</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">endpoints</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="background-color:#282828;color:#d69d85;">http</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/metrics</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">interval</span><span>: </span><span style="background-color:#282828;color:#d69d85;">30s</span><span>
</span></code></pre>
<p>Prometheus now automatically discovers and scrapes the genomics API every 30 seconds. No manual configuration when pods are rescheduled.</p>
<p><br><br></p>
<h2 id="sli-slo-sla-from-data-to-accountability">SLI, SLO, SLA: from data to accountability</h2>
<p>Collecting telemetry is pointless without defining what <em>good</em> looks like. <a href="https://sre.google/sre-book/service-level-objectives/">SLI, SLO, and SLA</a> are the framework for turning raw observability data into meaningful commitments.</p>
<p><strong><a href="https://sre.google/sre-book/service-level-objectives/#indicators-o8seIAcZ">SLI (Service Level Indicator)</a></strong> — a measurement. A specific, quantifiable signal that reflects the user experience:</p>
<blockquote>
<p><em>"The percentage of analysis API requests that return in under 2 seconds."</em></p>
</blockquote>
<p><strong><a href="https://sre.google/sre-book/service-level-objectives/#objectives-g0s1tdzt">SLO (Service Level Objective)</a></strong> — the target you set for that indicator:</p>
<blockquote>
<p><em>"99% of analysis requests should return in under 2 seconds, measured over a rolling 30-day window."</em></p>
</blockquote>
<p><strong><a href="https://sre.google/sre-book/service-level-objectives/#agreements-ub9SGXig">SLA (Service Level Agreement)</a></strong> — the formal commitment to external parties, with consequences:</p>
<blockquote>
<p><em>"If availability drops below 99.5% in any calendar month, affected institutions receive a service credit."</em></p>
</blockquote>
<p>The gap between your SLO (99%) and 100% is your <strong><a href="https://sre.google/sre-book/embracing-risk/#id-n9ITyhxZ">error budget</a></strong> — the room you have to take risks, deploy changes, and absorb incidents without breaching your commitment. If you have a 1% error budget over 30 days, that's about 7 hours of degraded service you can afford. Spend it wisely.</p>
<h3 id="measuring-the-sli-in-prometheus">Measuring the SLI in Prometheus</h3>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">groups</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-api.sli</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">record</span><span>: </span><span style="background-color:#282828;color:#d69d85;">job:genomics_api_p99_latency_seconds</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">expr</span><span>: </span><span style="color:#569cd6;">|
</span><span style="background-color:#282828;color:#d69d85;"> histogram_quantile(0.99,
</span><span style="background-color:#282828;color:#d69d85;"> rate(django_http_requests_latency_seconds_by_view_method_bucket{
</span><span style="background-color:#282828;color:#d69d85;"> job="genomics-api",
</span><span style="background-color:#282828;color:#d69d85;"> view="run_variant_analysis"
</span><span style="background-color:#282828;color:#d69d85;"> }[5m])
</span><span style="background-color:#282828;color:#d69d85;"> )
</span></code></pre>
<h3 id="alerting-when-the-error-budget-burns-too-fast">Alerting when the error budget burns too fast</h3>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">groups</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-api.slo</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">alert</span><span>: </span><span style="background-color:#282828;color:#d69d85;">AnalysisAPIErrorBudgetBurn</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">expr</span><span>: </span><span style="color:#569cd6;">|
</span><span style="background-color:#282828;color:#d69d85;"> (
</span><span style="background-color:#282828;color:#d69d85;"> rate(django_http_responses_total_by_status_view_method_total{
</span><span style="background-color:#282828;color:#d69d85;"> job="genomics-api",
</span><span style="background-color:#282828;color:#d69d85;"> status=~"5.."
</span><span style="background-color:#282828;color:#d69d85;"> }[1h])
</span><span style="background-color:#282828;color:#d69d85;"> /
</span><span style="background-color:#282828;color:#d69d85;"> rate(django_http_responses_total_by_status_view_method_total{
</span><span style="background-color:#282828;color:#d69d85;"> job="genomics-api"
</span><span style="background-color:#282828;color:#d69d85;"> }[1h])
</span><span style="background-color:#282828;color:#d69d85;"> ) > 0.01
</span><span> </span><span style="background-color:#282828;color:#569cd6;">for</span><span>: </span><span style="background-color:#282828;color:#d69d85;">5m</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">severity</span><span>: </span><span style="background-color:#282828;color:#d69d85;">warning</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">team</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-platform</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">annotations</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">summary</span><span>: </span><span style="color:#d69d85;">"Genomics API error rate above SLO threshold"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">description</span><span>: </span><span style="color:#569cd6;">>
</span><span style="background-color:#282828;color:#d69d85;"> Error rate is {{ $value | humanizePercentage }}.
</span><span style="background-color:#282828;color:#d69d85;"> At this rate, the monthly error budget will be exhausted in
</span><span style="background-color:#282828;color:#d69d85;"> {{ printf "%.1f" (div 0.01 $value | mul 720) }} hours.
</span></code></pre>
<p>This alert doesn't just tell you <em>that</em> the error rate is high — it tells you how quickly you are burning through your error budget, so you can decide whether to roll back immediately or investigate first.</p>
<p>A good alerting rule alerts on <strong>user impact</strong>, not on infrastructure symptoms. "Error budget burning at 10x" is more actionable than "CPU at 78%".</p>
<p><br><br></p>
<h2 id="conclusion">Conclusion</h2>
<p>A deployed system that you can't observe is a system you don't fully control. Logs tell you what happened. Metrics tell you how much and how often. Traces tell you where exactly things went wrong. Kubernetes events tell you what the cluster itself was doing. Together, they answer the question that every production incident eventually raises: <em>"what on earth is going on in there?"</em></p>
<p>The open-source stack — Prometheus, Grafana, Loki, Jaeger, OpenTelemetry — gives you all of this without vendor lock-in. It runs on EKS today, on bare metal tomorrow, on a laptop for local debugging when you need it. The investment in learning these tools pays dividends regardless of which cloud you're on.</p>
<p>And SLI/SLO/SLA takes that data one step further: it turns raw signals into a shared language between the people who build the system and the people who depend on it. Error budgets make the trade-off between reliability and velocity explicit, instead of leaving it as a permanent source of tension.</p>
<p>The platform is now provisioned, secured, deployed, and observable. That's a complete stack — and the knowledge to rebuild it anywhere.</p>
<br>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p><strong>Official documentation:</strong></p>
<ul>
<li><a href="https://opentelemetry.io/docs/">OpenTelemetry documentation</a></li>
<li><a href="https://opentelemetry-python.readthedocs.io/">OpenTelemetry Python SDK</a></li>
<li><a href="https://prometheus.io/docs/introduction/overview/">Prometheus documentation</a></li>
<li><a href="https://grafana.com/docs/">Grafana documentation</a></li>
<li><a href="https://grafana.com/docs/loki/latest/">Loki documentation</a></li>
<li><a href="https://www.jaegertracing.io/docs/">Jaeger documentation</a></li>
<li><a href="https://prometheus.io/docs/alerting/latest/alertmanager/">Prometheus Alertmanager</a></li>
</ul>
<p><strong>Tools:</strong></p>
<ul>
<li><a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">kube-prometheus-stack Helm chart</a> — Prometheus + Grafana + Alertmanager in one chart</li>
<li><a href="https://github.com/korfuri/django-prometheus">django-prometheus</a> — Prometheus metrics for Django</li>
<li><a href="https://pypi.org/project/opentelemetry-instrumentation-django/">opentelemetry-instrumentation-django</a> — auto-instrumentation for Django</li>
<li><a href="https://sre.google/sre-book/service-level-objectives/">Google SRE book — SLI/SLO/SLA chapter</a> — the canonical reference</li>
</ul>
<p><strong>Video tutorials:</strong></p>
<ul>
<li><a href="https://www.youtube.com/watch?v=r8UvWSX3KA8">OpenTelemetry explained in 10 minutes</a></li>
<li><a href="https://www.youtube.com/watch?v=QoDqxm7ybLc">Prometheus & Grafana on Kubernetes</a></li>
<li><a href="https://www.youtube.com/watch?v=EW9GMAWsyQU">Distributed tracing with Jaeger</a></li>
<li><a href="https://www.youtube.com/watch?v=tEylFyxbDLE">SLOs, SLAs, and Error Budgets</a></li>
</ul>
AWS (Day 6)2026-02-09T00:00:00+00:002026-02-09T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-6/ <img style="display: block; margin: 0 auto; width: 600px" src="/images/helm.gif" alt="Helm" title="Helm"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>This article is <strong>educational content</strong>. The examples are intentionally simplified for clarity. Before using these patterns in production, consult the <a href="https://developer.hashicorp.com/terraform/docs">Terraform documentation</a>, the <a href="https://helm.sh/docs/">Helm documentation</a>, and your platform team.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece. If you catch any errors, please reach out — I'd be grateful for the feedback and happy to make updates!</p>
</li>
</ol>
<br>
<br>
<h2 id="hook">Hook</h2>
<p><em>"We need to set up a new environment for the epidemiology team. Can you have something running by yesterday?"</em> That question is why this article exists: <em>how to easily repeat, redo, reiterate, duplicate, replicate, recreate, remake, reduplicate, the infrastructure we just built?</em></p>
<p>The previous episodes were focused on the theory: <a href="https://nskm.xyz/posts/aws-1/">VPCs and compute</a>, <a href="https://nskm.xyz/posts/aws-2-3/">IAM and encryption</a>, <a href="https://nskm.xyz/posts/aws-4/">Kubernetes and EKS</a>, <a href="https://nskm.xyz/posts/aws-5/">cluster security</a>. A colleague who missed the training would get the cluster up by hand: click through the console, type commands, copy-paste YAML files, and call it done. And it would <em>work</em>. Until he had to do it again for staging, for recette and again for production, and then a third time months later for a deployment in another country.</p>
<p>Deploying <a href="https://nskm.xyz/posts/syndromic-surveillance/">4s</a> for a new lab for a new country inside West Africa is something we've done many time in the last couple of months. Hopefully, deploying 4s <a href="https://www.youtube.com/watch?v=avbNOjCOHJE">will be done</a> for all the countries in Africa. Let me present to you 2 tools to solve that problem at two different layers: <strong>Terraform</strong> for provisioning the infrastructure, <strong>Helm</strong> for deploying the applications that run on it.</p>
<br>
<br>
<h2 id="toc">ToC</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-6/#the-problem-reproducibility-at-two-layers">The problem: reproducibility at two layers</a></li>
<li><a href="https://nskm.xyz/posts/aws-6/#terraform-infrastructure-as-code">Terraform: infrastructure as code</a></li>
<li><a href="https://nskm.xyz/posts/aws-6/#helm-kubernetes-package-manager">Helm: Kubernetes package manager</a></li>
<li><a href="https://nskm.xyz/posts/aws-6/#where-does-the-chart-live-chart-registries">Where does the chart live? Chart registries</a></li>
<li><a href="https://nskm.xyz/posts/aws-6/#putting-it-together-one-pipeline">Putting it together</a></li>
<li><a href="https://nskm.xyz/posts/aws-6/#more-on-this-topic">More on this topic</a></li>
</ol>
<p><br><br></p>
<h2 id="the-problem-reproducibility-at-two-layers">The problem: reproducibility at two layers</h2>
<p>Imagine you've just finished reading <a href="https://nskm.xyz/posts/aws-5/">Day 5</a>. You know how to secure an EKS cluster. Now you need to actually create one and deploy a Django REST API onto it.</p>
<p>You could do it manually, click through the EKS console, create a node group, write a few dozen YAML manifests, copy-paste the right <code>kubectl apply</code> commands. But we could do better. We should be able to easily redo what we've done.</p>
<p>Before we even get to the issue of reproducibility, we want to record, keep and preserve what has been done, if only so that we can review it later or go back and read through what has been achieved. As for me, throughout the workshop, I didn’t just type in commands; I wrote Bash scripts....</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/bash
</span><span>
</span><span style="color:#608b4e;"># Environment Variables
</span><span>AWS_REGION=</span><span style="background-color:#282828;color:#d69d85;">"eu-west-1"</span><span>
</span><span>CLUSTER_NAME=</span><span style="background-color:#282828;color:#d69d85;">"ptrck-foundation-training"</span><span>
</span><span>ACCOUNT_ID=</span><span style="background-color:#282828;color:#d69d85;">"546732826958"</span><span> </span><span style="color:#608b4e;"># ID of the EKS Admin role
</span><span>ROLE_ARN=</span><span style="background-color:#282828;color:#d69d85;">"arn:aws:iam::${</span><span style="background-color:#282828;color:#dcdcdc;">ACCOUNT_ID</span><span style="background-color:#282828;color:#d69d85;">}:role/EKSAdminRole"</span><span> </span><span style="color:#608b4e;"># ARN of the EKS Admin role
</span><span>CLUSTER_ARN=</span><span style="background-color:#282828;color:#d69d85;">"arn:aws:eks:${</span><span style="background-color:#282828;color:#dcdcdc;">AWS_REGION</span><span style="background-color:#282828;color:#d69d85;">}:${</span><span style="background-color:#282828;color:#dcdcdc;">ACCOUNT_ID</span><span style="background-color:#282828;color:#d69d85;">}:cluster/${</span><span style="background-color:#282828;color:#dcdcdc;">CLUSTER_NAME</span><span style="background-color:#282828;color:#d69d85;">}"</span><span>
</span><span>
</span><span style="color:#608b4e;"># I need to create this ECR repository
</span><span>ECR_REPO=</span><span style="background-color:#282828;color:#d69d85;">"ecr-repo-ptrck"</span><span>
</span><span>DEMO_NS=</span><span style="background-color:#282828;color:#d69d85;">"demo-ns-ptrck"</span><span>
</span><span>DEMO_DEPLOYMENT=</span><span style="background-color:#282828;color:#d69d85;">"demo-dply-ptrck"</span><span>
</span><span>DEMO_CONTAINER=</span><span style="background-color:#282828;color:#d69d85;">"app-ptrck"</span><span>
</span><span>
</span><span style="color:#608b4e;"># Helper function to print things with some separation
</span><span>section() {
</span><span> echo </span><span style="color:#d69d85;">""
</span><span> echo </span><span style="color:#d69d85;">"=====> $</span><span>1</span><span style="color:#d69d85;">"
</span><span>}
</span><span>
</span><span>
</span><span>section </span><span style="color:#d69d85;">"Configure AWS CLI credentials"
</span><span>aws configure
</span><span>
</span><span style="color:#608b4e;"># Identity & role assume
</span><span>section </span><span style="color:#d69d85;">"Verify identity"
</span><span>aws sts get-caller-identity
</span><span>
</span><span>section </span><span style="color:#d69d85;">"Verify role exists"
</span><span>aws iam get-role --role-name EKSAdminRole
</span><span>
</span><span>section </span><span style="color:#d69d85;">"List all roles in account (if you have permissions)"
</span><span>aws iam list-roles \
</span><span> --query </span><span style="color:#d69d85;">'Roles[].RoleName' </span><span>\
</span><span> --output text
</span><span>
</span><span>section </span><span style="color:#d69d85;">"List user Policies"
</span><span>aws iam list-user-policies --user-name $(aws sts get-caller-identity --query </span><span style="color:#d69d85;">'Arn'</span><span> --output text </span><span style="color:#569cd6;">| </span><span>cut -d</span><span style="color:#d69d85;">'/'</span><span> -f 6)
</span><span>
</span><span>section </span><span style="color:#d69d85;">"List attached policies"
</span><span>CURRENT_USER=</span><span style="background-color:#282828;color:#d69d85;">$(</span><span style="background-color:#282828;color:#dcdcdc;">aws</span><span style="background-color:#282828;color:#d69d85;"> sts get-caller-identity</span><span style="background-color:#282828;color:#dcdcdc;"> --query </span><span style="background-color:#282828;color:#d69d85;">'Arn'</span><span style="background-color:#282828;color:#dcdcdc;"> --output</span><span style="background-color:#282828;color:#d69d85;"> text </span><span style="background-color:#282828;color:#569cd6;">| </span><span style="background-color:#282828;color:#dcdcdc;">cut -d</span><span style="background-color:#282828;color:#d69d85;">'/'</span><span style="background-color:#282828;color:#dcdcdc;"> -f</span><span style="background-color:#282828;color:#d69d85;"> 6)</span><span>
</span><span>aws iam list-attached-user-policies --user-name $CURRENT_USER
</span><span>
</span><span>section </span><span style="color:#d69d85;">"Check current default region"
</span><span>aws configure get region
</span><span>
</span><span>section </span><span style="color:#d69d85;">"(Re)configure AWS CLI credentials"
</span><span>aws configure
</span><span>
</span><span>echo </span><span style="color:#d69d85;">""
</span><span>section </span><span style="color:#d69d85;">"Assume EKS admin role & retrieve the token"
</span><span>aws sts assume-role \
</span><span> --role-session-name iam-eks-admin-role \
</span><span> </span><span style="color:#608b4e;"># was working nice until sunday 7PM ?!
</span><span> --role-arn </span><span style="color:#d69d85;">"$</span><span>ROLE_ARN</span><span style="color:#d69d85;">" </span><span>\
</span><span> --duration-seconds 43200
</span><span>
</span><span>section </span><span style="color:#d69d85;">"Configure the 'iam-eks-admin-role' profile with the temporary credentials"
</span><span>aws configure --profile iam-eks-admin-role
</span><span>
</span><span>section </span><span style="color:#d69d85;">"What are the available roles ?"
</span><span>aws iam list-roles
</span><span>
</span><span>section </span><span style="color:#d69d85;">"What are the available policies?"
</span><span>aws iam list-policies
</span><span>
</span><span style="color:#608b4e;"># Check if there is a K8s cluster available
</span><span>section </span><span style="color:#d69d85;">"List all EKS clusters"
</span><span>aws eks list-clusters
</span><span>
</span><span>section </span><span style="color:#d69d85;">"List clusters in my region"
</span><span>aws eks list-clusters --region </span><span style="color:#d69d85;">"$</span><span>AWS_REGION</span><span style="color:#d69d85;">"
</span><span>
</span><span>section </span><span style="color:#d69d85;">"At this point, there is no clusters available whaaaaaa ?! lol"
</span><span>
</span><span>section </span><span style="color:#d69d85;">"Let's create our K8s cluster"
</span><span>eksctl create cluster \
</span><span> --name </span><span style="color:#d69d85;">"$</span><span>CLUSTER_NAME</span><span style="color:#d69d85;">" </span><span>\
</span><span> --region </span><span style="color:#d69d85;">"$</span><span>AWS_REGION</span><span style="color:#d69d85;">" </span><span>\
</span><span> --nodes 3 \
</span><span> --node-type t2.micro \
</span><span> </span><span style="color:#608b4e;"># --with-oidc: no need to separately call eksctl utils associate-iam-oidc-provider
</span><span> --with-oidc \
</span><span> --enable-auto-mode
</span></code></pre>
<p>This is the <strong>reproducibility problem</strong>, and it lives at two distinct layers:</p>
<ol>
<li>
<p><strong>Infrastructure layer</strong>: <em>Who created the VPC? What are the exact subnet CIDRs? Which KMS key encrypts etcd?</em> If the answer lives in someone's memory or a Confluence page, you have a problem.</p>
</li>
<li>
<p><strong>Application layer</strong>: <em>Which version of the Django app is running in staging? How do I roll back if the new release breaks something?</em> If the answer is "look at the kubectl commands in the README", you have a different problem.</p>
</li>
</ol>
<p><strong>Terraform</strong> solves the first. <strong>Helm</strong> solves the second.</p>
<p>Bash scripts are fine for exploration and one-off tasks, but they fall short once you need state tracking, idempotence, error handling, drift detection, and dependency management — which is exactly what infrastructure provisioning requires.</p>
<p><br><br></p>
<h2 id="terraform-infrastructure-as-code">Terraform: infrastructure as code</h2>
<h3 id="a-brief-history">A brief history</h3>
<p><a href="https://www.terraform.io/">Terraform</a> was created by <a href="https://www.hashicorp.com/">HashiCorp</a> and first released in <strong>2014</strong>. The problem it solved: AWS had CloudFormation, GCP had Deployment Manager, Azure had ARM templates — but if you used more than one cloud, you had three completely different tools, three syntaxes, three mental models. Terraform <a href="https://xkcd.com/927/">unified them</a> under a single declarative language: <strong>HCL</strong> (HashiCorp Configuration Language).</p>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/standards.png" alt="Standards" title="Standards"/>
<p>Core idea: <em>describe the desired state of your infrastructure, and Terraform figures out how to get there</em>. It maintains a <strong>state file</strong> tracking what it has created, so it can calculate the diff between current state and desired state on every run.</p>
<p>One important note: <em>as a free software advocate</em>, I should talk about the Terraform fork called <strong><a href="https://opentofu.org/">OpenTofu</a></strong>. OpenTofu is <a href="https://www.cncf.io/projects/opentofu/">CNCF</a> project, API-compatible with Terraform — same HCL, same providers, same workflow, <em>same everything</em>. For most teams, the choice between them is a licensing and governance question. The examples below work identically on both. <em>I hope.</em></p>
<h3 id="alternatives-why-terraform">Alternatives & why Terraform</h3>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Tool</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Approach</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">When to prefer it</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>AWS CloudFormation</strong></td>
<td style="border: 1px solid #333; padding: 10px;">AWS-native, JSON/YAML templates</td>
<td style="border: 1px solid #333; padding: 10px;">AWS-only shops that want native integration and no external tooling</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>AWS CDK</strong></td>
<td style="border: 1px solid #333; padding: 10px;">TypeScript/Python constructs that compile to CloudFormation</td>
<td style="border: 1px solid #333; padding: 10px;">Developer-heavy teams who want real programming languages for infrastructure</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Pulumi</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Multi-cloud, uses Python/TypeScript/Go directly</td>
<td style="border: 1px solid #333; padding: 10px;">Teams that find HCL limiting and want full programming language expressiveness</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Ansible</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Procedural YAML playbooks</td>
<td style="border: 1px solid #333; padding: 10px;">Configuration management and server setup; less suited for cloud resource provisioning</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>OpenTofu</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Terraform fork, identical syntax</td>
<td style="border: 1px solid #333; padding: 10px;">Teams that want open-source governance and no BSL licensing constraints</td>
</tr>
</tbody>
</table>
<p>Why we chose Terraform? It's multi-cloud (we're on AWS now, but that may change, lol), it has <a href="https://registry.terraform.io/browse/providers">over 4000 providers</a>, it has a mature module ecosystem, and it's the most widely adopted IaC tool in the industry.</p>
<h3 id="provisioning-an-eks-cluster">Provisioning an EKS cluster</h3>
<p>The goal: reproduce the EKS cluster from <a href="https://nskm.xyz/posts/aws-4/">Day 4</a> and <a href="https://nskm.xyz/posts/aws-5/">Day 5</a> — VPC, private subnets, encrypted etcd, managed node group — using Terraform. Every time. Identically.</p>
<p>Project structure:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>infra/
</span><span>├── main.tf # providers, backend, modules
</span><span>├── variables.tf # input variables
</span><span>├── outputs.tf # cluster name, endpoint (consumed by Helm later)
</span><span>└── terraform.tfvars # environment-specific values (not committed to Git)
</span></code></pre>
<p><strong><code>main.tf</code></strong> — the core:</p>
<pre data-lang="hcl" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-hcl "><code class="language-hcl" data-lang="hcl"><span>terraform {
</span><span> required_providers {
</span><span> aws = {
</span><span> source = "hashicorp/aws"
</span><span> version = "~> 5.0"
</span><span> }
</span><span> }
</span><span>
</span><span> # Remote state: shared, locked, versioned
</span><span> backend "s3" {
</span><span> bucket = "genomics-platform-tfstate"
</span><span> key = "eks/terraform.tfstate"
</span><span> region = "eu-west-3"
</span><span> dynamodb_table = "terraform-locks"
</span><span> encrypt = true
</span><span> }
</span><span>}
</span><span>
</span><span>provider "aws" {
</span><span> region = var.region
</span><span>}
</span><span>
</span><span># KMS key for etcd encryption (Day 5 best practice)
</span><span>resource "aws_kms_key" "eks" {
</span><span> description = "EKS secrets encryption"
</span><span> deletion_window_in_days = 7
</span><span>}
</span><span>
</span><span># VPC: private subnets for nodes, public subnets for load balancers
</span><span>module "vpc" {
</span><span> source = "terraform-aws-modules/vpc/aws"
</span><span> version = "~> 5.0"
</span><span>
</span><span> name = "${var.cluster_name}-vpc"
</span><span> cidr = "10.0.0.0/16"
</span><span>
</span><span> azs = ["${var.region}a", "${var.region}b", "${var.region}c"]
</span><span> private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
</span><span> public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
</span><span>
</span><span> enable_nat_gateway = true
</span><span> single_nat_gateway = true
</span><span>
</span><span> tags = {
</span><span> "kubernetes.io/cluster/${var.cluster_name}" = "shared"
</span><span> Environment = var.environment
</span><span> }
</span><span>}
</span><span>
</span><span># EKS cluster
</span><span>module "eks" {
</span><span> source = "terraform-aws-modules/eks/aws"
</span><span> version = "~> 20.0"
</span><span>
</span><span> cluster_name = var.cluster_name
</span><span> cluster_version = "1.30"
</span><span>
</span><span> vpc_id = module.vpc.vpc_id
</span><span> subnet_ids = module.vpc.private_subnets
</span><span>
</span><span> # Private endpoint only (Day 5 best practice)
</span><span> cluster_endpoint_public_access = false
</span><span>
</span><span> # etcd encryption with the KMS key above
</span><span> cluster_encryption_config = {
</span><span> resources = ["secrets"]
</span><span> provider_key_arn = aws_kms_key.eks.arn
</span><span> }
</span><span>
</span><span> eks_managed_node_groups = {
</span><span> genomics = {
</span><span> instance_types = ["m5.large"]
</span><span> min_size = 2
</span><span> max_size = 10
</span><span> desired_size = 3
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p><strong><code>variables.tf</code></strong>:</p>
<pre data-lang="hcl" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-hcl "><code class="language-hcl" data-lang="hcl"><span>variable "region" {
</span><span> description = "AWS region"
</span><span> type = string
</span><span> default = "eu-west-3"
</span><span>}
</span><span>
</span><span>variable "cluster_name" {
</span><span> description = "EKS cluster name"
</span><span> type = string
</span><span>}
</span><span>
</span><span>variable "environment" {
</span><span> description = "Deployment environment (dev, staging, prod)"
</span><span> type = string
</span><span>}
</span></code></pre>
<p><strong><code>outputs.tf</code></strong> — what other tools will need:</p>
<pre data-lang="hcl" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-hcl "><code class="language-hcl" data-lang="hcl"><span>output "cluster_name" {
</span><span> value = module.eks.cluster_name
</span><span>}
</span><span>
</span><span>output "cluster_endpoint" {
</span><span> value = module.eks.cluster_endpoint
</span><span>}
</span></code></pre>
<p>Deploy it:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>terraform init </span><span style="color:#608b4e;"># download providers and modules
</span><span>terraform plan </span><span style="color:#608b4e;"># preview what will be created
</span><span>terraform apply </span><span style="color:#608b4e;"># create the infrastructure
</span></code></pre>
<p>Voilà.</p>
<p><br><br></p>
<h2 id="helm-kubernetes-package-manager">Helm: Kubernetes package manager</h2>
<h3 id="a-brief-history-1">A brief history</h3>
<p><a href="https://helm.sh/">Helm</a> was first presented at <strong>KubeCon 2015</strong> by the team at <a href="https://deis.com/">Deis</a> (notably <a href="https://twitter.com/technosophos">Matt Butcher</a>). The problem it solved: writing raw Kubernetes manifests for a real application means dozens of YAML files, lots of copy-paste across environments, no concept of versioning, and no clean way to roll back. Helm introduced the <strong>chart</strong> — a package of Kubernetes manifests with templating and versioning built in.</p>
<p>In <strong>2016</strong>, Google and Deis merged their Kubernetes packaging work, and Helm was donated to the Kubernetes project. It became a <strong><a href="https://www.cncf.io/projects/helm/">CNCF graduated project</a></strong> in 2020 — the same tier as Kubernetes, Prometheus, and Envoy.</p>
<p>Helm 2 (the first widely adopted version) required a server-side component called <strong>Tiller</strong> running inside the cluster. Tiller had broad cluster-admin permissions and became a well-known security liability. <strong>Helm 3</strong> (released in 2019) removed Tiller entirely — all operations now happen client-side, and permissions come from your own kubeconfig. Simpler and significantly more secure.</p>
<h3 id="alternatives-why-helm">Alternatives & why Helm</h3>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Tool</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Approach</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">When to prefer it</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Kustomize</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Overlay-based patching, built into kubectl</td>
<td style="border: 1px solid #333; padding: 10px;">Simple apps where you want no templating and native kubectl support</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Plain manifests</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Raw YAML + <code>kubectl apply</code></td>
<td style="border: 1px solid #333; padding: 10px;">Very small projects, learning environments, one-off deployments</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Skaffold</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Dev-focused workflow: build + push + deploy</td>
<td style="border: 1px solid #333; padding: 10px;">Inner-loop development with fast local iteration cycles</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Carvel / ytt</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Structured YAML templating</td>
<td style="border: 1px solid #333; padding: 10px;">VMware/Tanzu ecosystems</td>
</tr>
</tbody>
</table>
<p>We chose Helm because: versioned releases let you see exactly what is running in each namespace with <code>helm list</code>, atomic upgrades roll back automatically on failure, <a href="https://helm.sh/docs/topics/charts_hooks/">hooks</a> let you run jobs before or after install (crucial for Django migrations — more on this below), and <a href="https://artifacthub.io/">ArtifactHub</a> hosts thousands of ready-made charts for databases, ingress controllers, monitoring stacks, and more. When you need to install cert-manager or the AWS Load Balancer Controller into your cluster, it's a one-liner.</p>
<h3 id="packaging-a-django-rest-api">Packaging a Django REST API</h3>
<p>Our syndromic surveillance platform is a Django REST API that processes public health data. Here's how to package it as a Helm chart.</p>
<p>Chart structure:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>syndromic-surveillance-api/
</span><span>├── Chart.yaml # chart metadata
</span><span>├── values.yaml # default configuration values
</span><span>└── templates/
</span><span> ├── _helpers.tpl # reusable template snippets
</span><span> ├── deployment.yaml
</span><span> ├── service.yaml
</span><span> └── migrate-job.yaml # Django migration hook
</span></code></pre>
<p><strong><code>Chart.yaml</code></strong>:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v2</span><span>
</span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">syndromic-surveillance-api</span><span>
</span><span style="background-color:#282828;color:#569cd6;">description</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Django REST API for genomic data processing</span><span>
</span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">application</span><span>
</span><span style="background-color:#282828;color:#569cd6;">version</span><span>: </span><span style="color:#b5cea8;">0.1.0 </span><span style="color:#608b4e;"># chart version — bump when the chart itself changes
</span><span style="background-color:#282828;color:#569cd6;">appVersion</span><span>: </span><span style="color:#d69d85;">"1.0.0" </span><span style="color:#608b4e;"># application version — overridden at deploy time
</span></code></pre>
<p><strong><code>values.yaml</code></strong> — the defaults, overridable per environment:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">replicaCount</span><span>: </span><span style="color:#b5cea8;">2
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">image</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">repository</span><span>: </span><span style="background-color:#282828;color:#d69d85;">123456789012.dkr.ecr.eu-west-3.amazonaws.com/syndromic-surveillance-api</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pullPolicy</span><span>: </span><span style="background-color:#282828;color:#d69d85;">IfNotPresent</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">tag</span><span>: </span><span style="color:#d69d85;">"" </span><span style="color:#608b4e;"># overridden at deploy time: --set image.tag=v1.2.3
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">service</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ClusterIP</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="color:#b5cea8;">8000
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">limits</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="background-color:#282828;color:#d69d85;">500m</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">512Mi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requests</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="background-color:#282828;color:#d69d85;">100m</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">256Mi</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DJANGO_SETTINGS_MODULE</span><span>: </span><span style="color:#d69d85;">"config.settings.production"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DB_HOST</span><span>: </span><span style="color:#d69d85;">"rds.syndromic-surveillance-platform.internal"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DB_NAME</span><span>: </span><span style="color:#d69d85;">"syndromic-surveillance"
</span></code></pre>
<p><strong><code>templates/deployment.yaml</code></strong>:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Deployment</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">include "syndromic-surveillance-api.fullname" .</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">include "syndromic-surveillance-api.labels" . | nindent 4</span><span> }}
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">replicas</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">.Values.replicaCount</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">include "syndromic-surveillance-api.selectorLabels" . | nindent 6</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">include "syndromic-surveillance-api.selectorLabels" . | nindent 8</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">securityContext</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runAsNonRoot</span><span>: </span><span style="color:#569cd6;">true </span><span style="color:#608b4e;"># Day 5 best practice
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runAsUser</span><span>: </span><span style="color:#b5cea8;">1000
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">.Chart.Name</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">"{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">containerPort</span><span>: </span><span style="color:#b5cea8;">8000
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">range $key</span><span>, </span><span style="background-color:#282828;color:#d69d85;">$val := .Values.env</span><span> }}
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">$key</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">value</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">$val | quote</span><span> }}
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">end</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">toYaml .Values.resources | nindent 10</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">readinessProbe</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">httpGet</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/healthz/</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="color:#b5cea8;">8000
</span><span> </span><span style="background-color:#282828;color:#569cd6;">initialDelaySeconds</span><span>: </span><span style="color:#b5cea8;">10
</span><span> </span><span style="background-color:#282828;color:#569cd6;">periodSeconds</span><span>: </span><span style="color:#b5cea8;">5
</span></code></pre>
<p>When, 2 years ago, I was first getting to grips with K8s, I remember using side containers to apply Django database migrations. Nowadays, I use “init containers” to do Django database migrations. And whilst doing some research on Helm, specifically to write <em>this article</em>, I was very surprised to discover that this thing exists. A <a href="https://helm.sh/docs/topics/charts_hooks/">pre-upgrade hook</a> that runs <code>manage.py migrate</code> <em>before</em> any new pods are started:</p>
<p><strong><code>templates/migrate-job.yaml</code></strong> — I really need to try this</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">batch/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Job</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">include "syndromic-surveillance-api.fullname" .</span><span> }}</span><span style="background-color:#282828;color:#d69d85;">-migrate</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">annotations</span><span>:
</span><span> </span><span style="color:#d69d85;">"helm.sh/hook"</span><span>: </span><span style="background-color:#282828;color:#d69d85;">pre-install,pre-upgrade</span><span>
</span><span> </span><span style="color:#d69d85;">"helm.sh/hook-delete-policy"</span><span>: </span><span style="background-color:#282828;color:#d69d85;">before-hook-creation</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">restartPolicy</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Never</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">securityContext</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runAsNonRoot</span><span>: </span><span style="color:#569cd6;">true
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runAsUser</span><span>: </span><span style="color:#b5cea8;">1000
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">migrate</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">"{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">command</span><span>: [</span><span style="color:#d69d85;">"python"</span><span>, </span><span style="color:#d69d85;">"manage.py"</span><span>, </span><span style="color:#d69d85;">"migrate"</span><span>, </span><span style="color:#d69d85;">"--noinput"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">range $key</span><span>, </span><span style="background-color:#282828;color:#d69d85;">$val := .Values.env</span><span> }}
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">$key</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">value</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">$val | quote</span><span> }}
</span><span> {{- </span><span style="background-color:#282828;color:#d69d85;">end</span><span> }}
</span></code></pre>
<p><em>Phew!</em></p>
<p>Deploy it:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># First deploy
</span><span>helm install syndromic-surveillance-api ./syndromic-surveillance-api \
</span><span> --namespace syndromic-surveillance \
</span><span> --create-namespace \
</span><span> --set image.tag=v1.0.0
</span><span>
</span><span style="color:#608b4e;"># Upgrade to a new version
</span><span>helm upgrade syndromic-surveillance-api ./syndromic-surveillance-api \
</span><span> --namespace syndromic-surveillance \
</span><span> --set image.tag=v1.2.3
</span><span>
</span><span style="color:#608b4e;"># Something broke — roll back to the previous release
</span><span>helm rollback syndromic-surveillance-api -n syndromic-surveillance
</span><span>
</span><span style="color:#608b4e;"># See what is currently running
</span><span>helm list -n syndromic-surveillance
</span></code></pre>
<h2 id="where-does-the-chart-live-chart-registries">Where does the chart live? Chart registries</h2>
<p>So far the chart lives in a local <code>./syndromic-surveillance-api/</code> directory. That works on your machine. It doesn't work for your teammates, your CI pipeline, or your staging environment.</p>
<p>Just like Docker images have registries (Docker Hub, ECR...), Helm charts have registries too.</p>
<p><strong>OCI registries — the recommended modern approach</strong></p>
<p>Since Helm 3.8 (2022), charts can be <a href="https://helm.sh/blog/storing-charts-in-oci/">stored as OCI artifacts</a>. This means the <em>same registry you already use for containers images</em> can store your charts. On AWS, a single ECR repository holds both:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># package and push the chart
</span><span>helm package ./syndromic-surveillance-api
</span><span>helm push syndromic-surveillance-api-0.1.0.tgz \
</span><span> oci://123456789012.dkr.ecr.eu-west-3.amazonaws.com/charts
</span><span>
</span><span style="color:#608b4e;"># teammates pull and install directly from ECR
</span><span>helm install syndromic-surveillance-api \
</span><span> oci://123456789012.dkr.ecr.eu-west-3.amazonaws.com/charts/syndromic-surveillance-api \
</span><span> --version 0.1.0 \
</span><span> --namespace syndromic-surveillance \
</span><span> --create-namespace
</span></code></pre>
<p><strong>What I recommend</strong></p>
<p>I decided to not talk about <strong>classic Helm repositories</strong> & <strong>public Helm registries</strong>. Since we are learning about the AWS cloud, just use ECR with OCI. You are already paying for it, IAM controls access (the same permissions model from <a href="https://nskm.xyz/posts/aws-2-3/">Day 2-3</a>), no extra infrastructure to maintain, and it works for both Docker images and Helm charts. Your teammates authenticate once via <code>aws ecr get-login-password</code> and pull charts the same way they pull images.</p>
<p><br><br></p>
<h2 id="putting-it-together-one-pipeline">Putting it together: one pipeline</h2>
<p>Terraform provisions the platform. Helm deploys the application. Here is the full sequence, end to end:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># 1. Provision the EKS cluster
</span><span>cd infra/
</span><span>terraform init
</span><span>terraform apply \
</span><span> -var=</span><span style="color:#d69d85;">"cluster_name=syndromic-surveillance-platform" </span><span>\
</span><span> -var=</span><span style="color:#d69d85;">"environment=staging"
</span><span>
</span><span style="color:#608b4e;"># 2. Configure kubectl to talk to the new cluster
</span><span>aws eks update-kubeconfig \
</span><span> --region eu-west-3 \
</span><span> --name syndromic-surveillance-platform
</span><span>
</span><span style="color:#608b4e;"># 3. Verify the nodes are ready
</span><span>kubectl get nodes
</span><span>
</span><span style="color:#608b4e;"># 4. Deploy the Django API
</span><span>cd ../syndromic-surveillance-api/
</span><span>helm upgrade --install syndromic-surveillance-api . \
</span><span> --namespace syndromic-surveillance \
</span><span> --create-namespace \
</span><span> --set image.tag=v1.0.0
</span><span>
</span><span style="color:#608b4e;"># 5. Verify
</span><span>kubectl get pods -n syndromic-surveillance
</span><span>helm list -n syndromic-surveillance
</span></code></pre>
<p>Five steps. New environment, new cluster, application running. The same sequence works for dev, staging, and prod — with a different <code>environment</code> variable and <code>image.tag</code> each time.</p>
<br>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>Two tools, one principle: <em>write once, run anywhere</em>. <em>(I’ve got some old memories coming back to me, lol – it was the Javaboyz who used to talk like that way back then in 1995).</em></p>
<p>Infrastructure and deployments should be like code: version-controlled, peer-reviewed, auditable, and repeatable. There is no alternative, we should adopt this way of working.</p>
<p>The learning curve is real. HCL takes time to learn. Helm templates can grow verbose. But the payoff is worth it every single time. As always, the subject is huge and cannot be covered in one tiny article. Here are some links you can use to learn more:</p>
<p><strong>Official documentation:</strong></p>
<ul>
<li><a href="https://developer.hashicorp.com/terraform/docs">Terraform documentation</a></li>
<li><a href="https://opentofu.org/docs/">OpenTofu documentation</a></li>
<li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs">Terraform AWS provider</a></li>
<li><a href="https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest">terraform-aws-modules/eks</a></li>
<li><a href="https://helm.sh/docs/">Helm documentation</a></li>
<li><a href="https://helm.sh/docs/topics/charts_hooks/">Helm chart hooks</a></li>
<li><a href="https://artifacthub.io/">ArtifactHub — public Helm chart repository</a></li>
<li><a href="https://chartmuseum.com/">ChartMuseum — self-hosted Helm chart repository</a></li>
<li><a href="https://helm.sh/docs/topics/registries/">Helm OCI support documentation</a></li>
</ul>
<p><strong>Tools:</strong></p>
<ul>
<li><a href="https://aquasecurity.github.io/tfsec/">tfsec</a> — static analysis for Terraform security misconfigurations</li>
<li><a href="https://www.infracost.io/">Infracost</a> — cost estimation for Terraform plans before you apply them</li>
<li><a href="https://github.com/norwoodj/helm-docs">helm-docs</a> — auto-generate documentation for Helm charts</li>
<li><a href="https://github.com/helm/chart-testing">chart-testing</a> — lint and test Helm charts in CI</li>
</ul>
<p><strong>Video tutorials:</strong></p>
<ul>
<li><a href="https://www.youtube.com/watch?v=fy8SHvNZGeE">What's Helm?</a></li>
<li><a href="https://www.youtube.com/watch?v=tomUWcQ0P3k">Terraform in 100 seconds</a></li>
<li><a href="https://www.youtube.com/watch?v=7xngnjfIlK4">Complete Terraform course</a></li>
<li><a href="https://www.youtube.com/watch?v=-ykwb1d0DXU">Helm and Kubernetes tutorial</a></li>
<li><a href="https://www.youtube.com/watch?v=MNt2HGxClZ0">Terraform EKS cluster from scratch</a></li>
</ul>
AWS (Day 5)2026-02-06T00:00:00+00:002026-02-06T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-5/ <img style="display: block; margin: 0 auto; width: 400px" src="/images/get_out.gif" alt="Security checkpoint" title="Security checkpoint"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>This article is <strong>educational content</strong>, not a production hardening guide. Before securing a real EKS cluster, consult the <a href="https://aws.github.io/aws-eks-best-practices/">AWS EKS best practices guide</a> and involve your security teams.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece, if you catch any errors, please reach out—I'd be grateful for the feedback and happy to make updates!</p>
</li>
</ol>
<br>
<br>
<h2 id="hook">Hook</h2>
<p>Today's topic feels like the natural conclusion of the entire week. We spent Day 1 building the <a href="https://nskm.xyz/posts/aws-1/">foundation</a>: regions, VPCs, compute, storage. Days 2-3 were about <a href="https://nskm.xyz/posts/aws-2-3/">who can do what and how to prove it</a>: IAM, OIDC, encryption, audit. Day 4 introduced <a href="https://nskm.xyz/posts/aws-4/">K8s and EKS</a>: the orchestration layer. Now we answer the question that matters most: <strong>how do you make sure nobody breaks into the thing you just built?</strong></p>
<p>Security in K8s is not a feature you turn on. It's a <a href="https://www.youtube.com/watch?v=_GgfLZoLuGY">layer</a> you build, piece by piece. And the uncomfortable truth is: K8s, out of the box, is <em>not</em> <a href="https://www.tigera.io/blog/kubernetes-is-powerful-but-not-secure-at-least-not-by-default/">secure</a>. It gives you the <em>mechanisms</em> — but the defaults are permissive, and the responsibility is yours.</p>
<p>Let's talk about what that means.</p>
<br>
<br>
<h2 id="table-of-contents">Table of contents</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-5/#k8s-security-concepts-traditional-infrastructure">K8s security concepts → traditional infrastructure</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#how-secure-is-k8s-by-default">How secure is K8s by default?</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#authentication-who-are-you">Authentication: who are you?</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#authorization-what-can-you-do">Authorization: what can you do?</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#network-security">Network security</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#pod-security">Pod security</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#secrets-management">Secrets management</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#resource-quotas-limitranges">Resource quotas & LimitRanges</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#audit-observability">Audit & observability</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#best-practices-checklist">Best practices checklist</a></li>
<li><a href="https://nskm.xyz/posts/aws-5/#more-on-this-topic">More on this topic</a></li>
</ol>
<p><br><br></p>
<h2 id="k8s-security-concepts-traditional-infrastructure">K8s security concepts → traditional infrastructure</h2>
<p>If you've been securing Linux servers, you already know these concepts. K8s just applies them at orchestration scale:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">K8s Security Concept</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Traditional Equivalent</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">What's Different?</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>RBAC (Roles & RoleBindings)</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Linux users, groups, sudo, <code>/etc/sudoers</code></td>
<td style="border: 1px solid #333; padding: 10px;">Scoped to namespaces or cluster-wide; applies to API verbs (get, create, delete), not file permissions</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Service Accounts</strong></td>
<td style="border: 1px solid #333; padding: 10px;">System users (<code>www-data</code>, <code>postgres</code>)</td>
<td style="border: 1px solid #333; padding: 10px;">Identity for pods, not humans; auto-mounted tokens; can map to cloud IAM roles</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Network Policies</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>iptables</code>, firewall rules</td>
<td style="border: 1px solid #333; padding: 10px;">Declarative YAML; enforced by the CNI plugin; scoped by pod labels and namespaces</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Pod Security Admission</strong></td>
<td style="border: 1px solid #333; padding: 10px;">AppArmor, SELinux, seccomp profiles</td>
<td style="border: 1px solid #333; padding: 10px;">Cluster-level enforcement of what containers can do (run as root, mount host paths, etc.)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Secrets</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>/etc/shadow</code>, SSH keys, environment variables</td>
<td style="border: 1px solid #333; padding: 10px;">Stored in etcd; base64-encoded (not encrypted!) by default; can integrate with external vaults</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Resource Quotas</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>ulimit</code>, cgroups, disk quotas</td>
<td style="border: 1px solid #333; padding: 10px;">Applied per namespace; limits CPU, memory, pod count, storage claims</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Audit Logging</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>auditd</code>, <code>/var/log/auth.log</code>, syslog</td>
<td style="border: 1px solid #333; padding: 10px;">API server records every request; configurable verbosity levels; integrates with cloud logging</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Namespaces</strong></td>
<td style="border: 1px solid #333; padding: 10px;">chroot, Linux namespaces, separate VMs</td>
<td style="border: 1px solid #333; padding: 10px;">Logical isolation only — not a security boundary by themselves without RBAC + Network Policies</td>
</tr>
</tbody>
</table>
<p>The mental model is: <strong>K8s security = Linux security concepts, applied declaratively, at the API level, across a fleet of containers.</strong></p>
<p><br><br></p>
<h2 id="how-secure-is-k8s-by-default">How secure is K8s by default?</h2>
<p>The short answer: <strong>not very</strong>. You have to actively tighten things.</p>
<h3 id="what-you-get-out-of-the-box">What you get out of the box</h3>
<ul>
<li><strong>API server authentication</strong>: The API server requires authentication — anonymous requests are rejected for most operations</li>
<li><strong>Namespaces</strong>: Logical separation exists from the start (<code>default</code>, <code>kube-system</code>, <code>kube-public</code>)</li>
<li><strong>etcd is only accessible from the control plane</strong>: In a properly set up cluster, etcd isn't exposed to worker nodes or external networks</li>
<li><strong>RBAC is enabled</strong>: The authorization mode includes RBAC (in modern K8s versions)</li>
</ul>
<h3 id="what-is-not-secure-by-default">What is NOT secure by default</h3>
<p><strong><a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">No network policies enforced.</a></strong> By default, every pod can talk to every other pod in the cluster. Your frontend pod can reach your database pod. Your staging namespace can reach your production namespace. It's like running all your servers in the same network segment with no firewall.</p>
<p><strong><a href="https://kubernetes.io/docs/concepts/configuration/secret/">Secrets are not encrypted at rest.</a></strong> K8s Secrets are stored as base64-encoded values in etcd. Base64 is <em>encoding</em>, not <em>encryption</em>. Anyone with access to etcd can read every secret in your cluster. This is like storing passwords in a text file and calling it security.</p>
<p><strong><a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/">Containers run as root by default.</a></strong> Unless you explicitly specify a security context, your container process runs as UID 0 — the same root as on the underlying node. A container escape vulnerability becomes a root-level compromise of the node.</p>
<p><strong><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/">Default service accounts are overly permissive.</a></strong> Every namespace gets a <code>default</code> service account that is automatically mounted into every pod. In many setups, this service account has more permissions than needed — and pods that don't need API access still get a token.</p>
<p><strong><a href="https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/">No audit logging enabled by default.</a></strong> The API server <em>can</em> log every request, but audit logging is not configured out of the box. Without it, you have no record of who did what in your cluster.</p>
<h3 id="the-eks-twist">The EKS twist</h3>
<p>AWS tightens some things for you when you use EKS:</p>
<ul>
<li><strong>etcd encryption at rest</strong> is available (you enable it with a KMS key — it's not automatic, but it's a one-click option)</li>
<li><strong>The control plane is managed</strong>: You can't SSH into it, and AWS handles patching — reducing your attack surface</li>
<li><strong>API server endpoint access</strong> can be restricted to your VPC (private endpoint)</li>
<li><strong>EKS control plane logging</strong> can be sent to CloudWatch (but you have to enable it)</li>
</ul>
<p>What remains your responsibility: network policies, pod security, RBAC configuration, secrets management, image scanning, and everything at the workload level. AWS secures the <em>infrastructure</em>; you secure the <em>workloads</em>.</p>
<p><br><br></p>
<h2 id="authentication-who-are-you">Authentication: who are you?</h2>
<p>Before K8s can decide what you're allowed to do, it needs to know <em>who you are</em>. There are two types of identities in K8s: human users and service accounts.</p>
<h3 id="k8s-service-accounts">K8s Service Accounts</h3>
<p>A <strong><a href="https://kubernetes.io/docs/concepts/security/service-accounts/">Service Account</a></strong> is an identity for processes running inside pods. Unlike human users (which K8s doesn't manage directly), service accounts are native K8s objects:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ServiceAccount</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-pipeline</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research</span><span>
</span></code></pre>
<p>Every namespace has a <code>default</code> service account. Every pod that doesn't specify a service account gets the <code>default</code> one, along with a mounted token at <code>/var/run/secrets/kubernetes.io/serviceaccount/token</code>.</p>
<p><strong>Best practice</strong>: Create dedicated service accounts for each workload and disable auto-mounting of tokens for pods that don't need API access:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ServiceAccount</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data-processor</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research</span><span>
</span><span style="background-color:#282828;color:#569cd6;">automountServiceAccountToken</span><span>: </span><span style="color:#569cd6;">false
</span></code></pre>
<h3 id="aws-iam-integration-with-eks">AWS IAM integration with EKS</h3>
<p>In EKS, human authentication flows through AWS IAM. When you run <code>kubectl</code> against an EKS cluster, this is what happens:</p>
<ol>
<li><strong>You authenticate to AWS</strong> (IAM user, role, or SSO)</li>
<li><strong><code>aws eks get-token</code></strong> generates a token based on your IAM identity</li>
<li><strong>The EKS API server</strong> validates this token against the AWS IAM authenticator</li>
<li><strong>Your IAM identity is mapped</strong> to a K8s user/group via the <code>aws-auth</code> ConfigMap or <a href="https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html">EKS access entries</a></li>
<li><strong>RBAC takes over</strong> — K8s decides what this user can do</li>
</ol>
<p>For pods that need to call AWS services (S3, DynamoDB, SQS), EKS offers two mechanisms:</p>
<ul>
<li><strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">IRSA (IAM Roles for Service Accounts)</a></strong>: We covered this in the <a href="https://nskm.xyz/posts/aws-4/">previous article</a>.</li>
<li><strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html">EKS Pod Identity</a></strong>: A newer mechanism that achieves the same thing without needing to configure an OIDC provider yourself.</li>
</ul>
<p>Both eliminate the need to store AWS credentials as K8s Secrets. Temporary credentials are injected automatically and rotated by AWS.</p>
<p>The identity chain looks like this: <strong>AWS IAM</strong> (authenticates the human or CI system) → <strong>K8s RBAC</strong> (authorizes what they can do inside the cluster) → <strong>IRSA/Pod Identity</strong> (gives pods AWS permissions without static credentials).</p>
<p><br><br></p>
<h2 id="authorization-what-can-you-do">Authorization: what can you do?</h2>
<p>Authentication tells K8s <em>who</em> you are. <strong><a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RBAC (Role-Based Access Control)</a></strong> tells K8s <em>what</em> you're allowed to do. It's the <code>sudo</code> and <code>/etc/sudoers</code> of the K8s world — except it operates on API verbs and resources instead of commands and files.</p>
<h3 id="the-four-rbac-objects">The four RBAC objects</h3>
<p>RBAC uses four objects, organized in two pairs:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Object</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Scope</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Purpose</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Role</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Namespace</td>
<td style="border: 1px solid #333; padding: 10px;">Defines <em>what</em> actions are allowed on <em>which</em> resources within a namespace</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>RoleBinding</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Namespace</td>
<td style="border: 1px solid #333; padding: 10px;">Binds a Role to a user, group, or service account</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>ClusterRole</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Cluster-wide</td>
<td style="border: 1px solid #333; padding: 10px;">Same as Role, but applies across all namespaces</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>ClusterRoleBinding</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Cluster-wide</td>
<td style="border: 1px solid #333; padding: 10px;">Binds a ClusterRole to a user, group, or service account</td>
</tr>
</tbody>
</table>
<h3 id="practical-example-biomedical-research-context">Practical example: biomedical research context</h3>
<p>Imagine a research cluster with two teams: one running genomic analysis pipelines, another managing a clinical trial data platform. You want each team to only access their own namespace:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># Role: genomics researchers can manage pods, services, and jobs in their namespace
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">rbac.authorization.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Role</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-developer</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">apiGroups</span><span>: [</span><span style="color:#d69d85;">""</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>: [</span><span style="color:#d69d85;">"pods"</span><span>, </span><span style="color:#d69d85;">"services"</span><span>, </span><span style="color:#d69d85;">"configmaps"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">verbs</span><span>: [</span><span style="color:#d69d85;">"get"</span><span>, </span><span style="color:#d69d85;">"list"</span><span>, </span><span style="color:#d69d85;">"watch"</span><span>, </span><span style="color:#d69d85;">"create"</span><span>, </span><span style="color:#d69d85;">"update"</span><span>, </span><span style="color:#d69d85;">"delete"</span><span>]
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">apiGroups</span><span>: [</span><span style="color:#d69d85;">"batch"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>: [</span><span style="color:#d69d85;">"jobs"</span><span>, </span><span style="color:#d69d85;">"cronjobs"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">verbs</span><span>: [</span><span style="color:#d69d85;">"get"</span><span>, </span><span style="color:#d69d85;">"list"</span><span>, </span><span style="color:#d69d85;">"watch"</span><span>, </span><span style="color:#d69d85;">"create"</span><span>, </span><span style="color:#d69d85;">"delete"</span><span>]
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">apiGroups</span><span>: [</span><span style="color:#d69d85;">""</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>: [</span><span style="color:#d69d85;">"secrets"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">verbs</span><span>: [</span><span style="color:#d69d85;">"get"</span><span>, </span><span style="color:#d69d85;">"list"</span><span>] </span><span style="color:#608b4e;"># Can read secrets but not create/modify them
</span></code></pre>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># RoleBinding: attach this role to the genomics team group
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">rbac.authorization.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">RoleBinding</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-team-binding</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span style="background-color:#282828;color:#569cd6;">subjects</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Group</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-team</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">apiGroup</span><span>: </span><span style="background-color:#282828;color:#d69d85;">rbac.authorization.k8s.io</span><span>
</span><span style="background-color:#282828;color:#569cd6;">roleRef</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Role</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-developer</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">apiGroup</span><span>: </span><span style="background-color:#282828;color:#d69d85;">rbac.authorization.k8s.io</span><span>
</span></code></pre>
<p>The clinical trials team gets a similar setup in their own namespace — and <em>cannot</em> see or touch anything in the <code>genomics</code> namespace. This is the same principle as giving each department its own Linux user group with permissions scoped to specific directories — except here the "directories" are K8s namespaces and the "permissions" are API operations.</p>
<br>
<h2 id="network-security">Network security</h2>
<p>By default, K8s is a flat network: every pod can talk to every other pod. This is convenient for development but a disaster for security. This is like every server in a data center on the same VLAN with no firewall.</p>
<h3 id="network-policies">Network Policies</h3>
<p><strong><a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">Network Policies</a></strong> are the <code>iptables</code> of the K8s world. They let you control which pods can communicate with which other pods.</p>
<p>A Network Policy selects pods using labels and defines allowed ingress (incoming) and/or egress (outgoing) traffic:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># Default deny all ingress traffic in the namespace
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">networking.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">NetworkPolicy</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">default-deny-ingress</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">clinical-trials</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">podSelector</span><span>: {} </span><span style="color:#608b4e;"># Applies to ALL pods in this namespace
</span><span> </span><span style="background-color:#282828;color:#569cd6;">policyTypes</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">Ingress</span><span>
</span></code></pre>
<p>This single manifest changes everything: now no pod in <code>clinical-trials</code> can receive traffic unless explicitly allowed. This is the "default deny" approach, like configuring a firewall to block everything, then opening specific ports.</p>
<p>Now allow specific traffic:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># Allow only the web frontend to talk to the API server
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">networking.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">NetworkPolicy</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">allow-frontend-to-api</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">clinical-trials</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">podSelector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">trials-api</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">policyTypes</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">Ingress</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ingress</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">from</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">podSelector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">trials-frontend</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">protocol</span><span>: </span><span style="background-color:#282828;color:#d69d85;">TCP</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="color:#b5cea8;">8080
</span></code></pre>
<p><strong>Important caveat</strong>: Network Policies only work if your CNI plugin supports them. Calico, Cilium, and Weave support them. The default AWS VPC CNI <em>does not</em> enforce Network Policies natively — you need to install <a href="https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks">Calico</a> or enable the <a href="https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html">AWS Network Policy Controller</a> (available since EKS v1.25 with VPC CNI v1.14+).</p>
<h3 id="security-groups-for-pods-eks-specific">Security Groups for Pods (EKS-specific)</h3>
<p>In EKS, you can also apply <strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html">Security Groups directly to pods</a></strong>. This bridges the K8s and AWS networking worlds: your pods get the same Security Group enforcement as EC2 instances.</p>
<p>This is useful when you need your pods to interact with AWS resources that use Security Groups for access control (like RDS databases). Instead of opening the database to the entire node's Security Group, you assign a specific Security Group to only the pods that need database access.</p>
<h3 id="comparison-network-policies-vs-security-groups-vs-nacls">Comparison: Network Policies vs Security Groups vs NACLs</h3>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Feature</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">K8s Network Policies</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">AWS Security Groups</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">AWS NACLs</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Scope</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Pod-to-pod within cluster</td>
<td style="border: 1px solid #333; padding: 10px;">ENI-level (instance or pod)</td>
<td style="border: 1px solid #333; padding: 10px;">Subnet-level</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Stateful?</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Depends on CNI</td>
<td style="border: 1px solid #333; padding: 10px;">Yes (return traffic auto-allowed)</td>
<td style="border: 1px solid #333; padding: 10px;">No (must allow both directions)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Default behavior</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Allow all (until a policy is applied)</td>
<td style="border: 1px solid #333; padding: 10px;">Deny all inbound, allow all outbound</td>
<td style="border: 1px solid #333; padding: 10px;">Allow all (default NACL)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Selection method</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Pod labels, namespace selectors</td>
<td style="border: 1px solid #333; padding: 10px;">Attached to ENIs</td>
<td style="border: 1px solid #333; padding: 10px;">Applied to subnets</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Aware of K8s concepts?</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Yes (pods, namespaces, labels)</td>
<td style="border: 1px solid #333; padding: 10px;">No (IP addresses and ports only)</td>
<td style="border: 1px solid #333; padding: 10px;">No (IP ranges and ports only)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Best for</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Intra-cluster segmentation</td>
<td style="border: 1px solid #333; padding: 10px;">Controlling access to AWS resources</td>
<td style="border: 1px solid #333; padding: 10px;">Subnet-level guardrails</td>
</tr>
</tbody>
</table>
<p>In practice, you use <strong>all three layers together</strong>. Network Policies for pod-to-pod rules, Security Groups for pod-to-AWS-resource rules, and NACLs as a coarse subnet-level safety net. Defense in depth.</p>
<img style="display: block; margin: 0 auto; width: 600px" src="/images/eks_security_groups_pod_mapping.svg" alt="" title="I am obliged to admit I'm impressed by this level of orchestration"/>
<hr />
<p><br><br></p>
<h2 id="pod-security">Pod security</h2>
<p>Even if your network is locked down, a compromised container running as root with full Linux capabilities is a serious problem. Pod security is about restricting <em>what containers can do</em> on the node.</p>
<h3 id="pod-security-standards-pod-security-admission-psa">Pod Security Standards & Pod Security Admission (PSA)</h3>
<p>K8s defines three <strong><a href="https://kubernetes.io/docs/concepts/security/pod-security-standards/">Pod Security Standards</a></strong> — profiles that describe increasing levels of restriction:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Profile</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Description</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Use case</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Privileged</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Unrestricted. No security checks.</td>
<td style="border: 1px solid #333; padding: 10px;">System-level workloads (CNI plugins, logging agents)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Baseline</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Prevents known privilege escalations. Blocks hostNetwork, hostPID, privileged containers.</td>
<td style="border: 1px solid #333; padding: 10px;">Most application workloads</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Restricted</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Heavily restricted. Must run as non-root, drop all capabilities, read-only root filesystem.</td>
<td style="border: 1px solid #333; padding: 10px;">Security-sensitive workloads</td>
</tr>
</tbody>
</table>
<p><strong><a href="https://kubernetes.io/docs/concepts/security/pod-security-admission/">Pod Security Admission (PSA)</a></strong> enforces these standards at the namespace level using labels:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Namespace</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">clinical-trials</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pod-security.kubernetes.io/enforce</span><span>: </span><span style="background-color:#282828;color:#d69d85;">restricted</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pod-security.kubernetes.io/warn</span><span>: </span><span style="background-color:#282828;color:#d69d85;">restricted</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pod-security.kubernetes.io/audit</span><span>: </span><span style="background-color:#282828;color:#d69d85;">restricted</span><span>
</span></code></pre>
<p>With this configuration, any pod in <code>clinical-trials</code> that violates the <code>restricted</code> profile will be rejected. The <code>warn</code> and <code>audit</code> modes give you visibility before you enforce.</p>
<h3 id="securitycontext">SecurityContext</h3>
<p>For fine-grained control, every pod and container can specify a <strong><a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/">SecurityContext</a></strong>:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Pod</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">secure-analysis-job</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">securityContext</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runAsNonRoot</span><span>: </span><span style="color:#569cd6;">true
</span><span> </span><span style="background-color:#282828;color:#569cd6;">runAsUser</span><span>: </span><span style="color:#b5cea8;">1000
</span><span> </span><span style="background-color:#282828;color:#569cd6;">fsGroup</span><span>: </span><span style="color:#b5cea8;">2000
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">analyzer</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research-tools:v3</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">securityContext</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">allowPrivilegeEscalation</span><span>: </span><span style="color:#569cd6;">false
</span><span> </span><span style="background-color:#282828;color:#569cd6;">readOnlyRootFilesystem</span><span>: </span><span style="color:#569cd6;">true
</span><span> </span><span style="background-color:#282828;color:#569cd6;">capabilities</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">drop</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ALL</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumeMounts</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">tmp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">mountPath</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/tmp</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">results</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">mountPath</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/data/results</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">tmp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">emptyDir</span><span>: {}
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">results</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">persistentVolumeClaim</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">claimName</span><span>: </span><span style="background-color:#282828;color:#d69d85;">analysis-results</span><span>
</span></code></pre>
<p>This pod: runs as a non-root user (UID 1000), cannot escalate privileges, has a read-only root filesystem (with writable <code>/tmp</code> and <code>/data/results</code> mounts), and drops all Linux capabilities. A compromised container in this configuration can do <em>very</em> little damage.</p>
<h3 id="workload-isolation-via-scheduling-constraints">Workload isolation via scheduling constraints</h3>
<p>For sensitive workloads that should not share nodes with untrusted workloads, you can use <strong>taints and tolerations</strong> along with <strong>node affinity</strong>:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># Taint a node group for sensitive workloads only
</span><span style="color:#608b4e;"># kubectl taint nodes -l workload=sensitive sensitive=true:NoSchedule
</span></code></pre>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># Pod that tolerates the taint and prefers sensitive nodes
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">tolerations</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">key</span><span>: </span><span style="color:#d69d85;">"sensitive"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">operator</span><span>: </span><span style="color:#d69d85;">"Equal"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">value</span><span>: </span><span style="color:#d69d85;">"true"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">effect</span><span>: </span><span style="color:#d69d85;">"NoSchedule"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">affinity</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">nodeAffinity</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requiredDuringSchedulingIgnoredDuringExecution</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">nodeSelectorTerms</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">matchExpressions</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">key</span><span>: </span><span style="background-color:#282828;color:#d69d85;">workload</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">operator</span><span>: </span><span style="background-color:#282828;color:#d69d85;">In</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">values</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">sensitive</span><span>
</span></code></pre>
<p>This is not full multi-tenancy (K8s isn't designed for hostile multi-tenancy), but it ensures that your genomic analysis pipelines don't share hardware with less trusted workloads.</p>
<p><br><br></p>
<h2 id="secrets-management">Secrets management</h2>
<p>Every application has secrets: database passwords, API keys, TLS certificates. How you manage them in K8s matters <em>a lot</em>.</p>
<h3 id="k8s-secrets-and-why-they-re-not-really-secret">K8s Secrets (and why they're not really secret)</h3>
<p>K8s has a built-in Secret object:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Secret</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">db-credentials</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research</span><span>
</span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Opaque</span><span>
</span><span style="background-color:#282828;color:#569cd6;">data</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">username</span><span>: </span><span style="background-color:#282828;color:#d69d85;">cG9zdGdyZXM=</span><span> </span><span style="color:#608b4e;"># base64 of "postgres"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">password</span><span>: </span><span style="background-color:#282828;color:#d69d85;">c3VwZXJzZWNyZXQ=</span><span> </span><span style="color:#608b4e;"># base64 of "supersecret"
</span></code></pre>
<p>The problem: this is base64 <em>encoding</em>, not encryption. Run <code>echo c3VwZXJzZWNyZXQ= | base64 -d</code> and you get <code>supersecret</code>. Anyone with <code>kubectl get secret -o yaml</code> access can read all your secrets. And in etcd, they're stored in plain text (well, base64).</p>
<p>K8s Secrets are useful as <em>a delivery mechanism</em> (mounting credentials into pods), but they are <strong>not a secret store</strong>.</p>
<h3 id="encryption-at-rest-with-kms">Encryption at rest with KMS</h3>
<p>The minimum you should do: enable <strong><a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/">encryption at rest for etcd</a></strong>. In EKS, this is straightforward — you provide a KMS key when creating or updating the cluster:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>aws eks create-cluster \
</span><span> --name my-cluster \
</span><span> --encryption-config </span><span style="color:#d69d85;">'[{
</span><span style="color:#d69d85;"> "resources": ["secrets"],
</span><span style="color:#d69d85;"> "provider": {
</span><span style="color:#d69d85;"> "keyArn": "arn:aws:kms:eu-west-3:123456789012:key/your-kms-key-id"
</span><span style="color:#d69d85;"> }
</span><span style="color:#d69d85;"> }]' </span><span>\
</span><span> ...
</span></code></pre>
<p>Now secrets are encrypted in etcd using your KMS key. This protects against someone gaining access to the underlying storage — but anyone with RBAC permissions to <code>get secrets</code> can still read them through the API.</p>
<h3 id="external-secret-stores">External secret stores</h3>
<p>For production, you should integrate with a dedicated secret management solution:</p>
<p><strong><a href="https://aws.amazon.com/secrets-manager/">AWS Secrets Manager</a></strong> or <strong><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html">AWS Systems Manager Parameter Store</a></strong>: Store secrets in AWS, fetch them into K8s at runtime. The <a href="https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html">AWS Secrets Store CSI Driver</a> mounts AWS secrets as volumes in your pods.</p>
<p><strong><a href="https://external-secrets.io/">External Secrets Operator</a></strong>: A K8s operator that syncs secrets from external stores (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) into K8s Secret objects. You define an <code>ExternalSecret</code> resource, and the operator keeps the K8s Secret in sync:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">external-secrets.io/v1beta1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ExternalSecret</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">db-credentials</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">refreshInterval</span><span>: </span><span style="background-color:#282828;color:#d69d85;">1h</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">secretStoreRef</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">aws-secrets-manager</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ClusterSecretStore</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">target</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">db-credentials</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">data</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">secretKey</span><span>: </span><span style="background-color:#282828;color:#d69d85;">username</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">remoteRef</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">key</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research/db-credentials</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">property</span><span>: </span><span style="background-color:#282828;color:#d69d85;">username</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">secretKey</span><span>: </span><span style="background-color:#282828;color:#d69d85;">password</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">remoteRef</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">key</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research/db-credentials</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">property</span><span>: </span><span style="background-color:#282828;color:#d69d85;">password</span><span>
</span></code></pre>
<p><strong><a href="https://sealed-secrets.netlify.app/">Sealed Secrets</a></strong>: Encrypts secrets so they can be safely stored in Git. A controller in the cluster decrypts them. Useful for GitOps workflows where you want everything in version control but can't commit plain secrets.</p>
<p>The pattern is: <strong>secrets live in a dedicated, encrypted, access-controlled store</strong> (AWS Secrets Manager, Vault); <strong>K8s fetches them at runtime</strong> (via CSI driver or operator); <strong>RBAC limits who can access them</strong> within the cluster.</p>
<p><br><br></p>
<h2 id="resource-quotas-limitranges">Resource quotas & LimitRanges</h2>
<p>This is the security concern people forget: <strong>resource abuse</strong>. A single runaway pod consuming all CPU or memory on a node is a denial-of-service attack — even if it's accidental. In a shared cluster (multiple teams, multiple namespaces), resource guardrails are essential.</p>
<h3 id="resourcequotas">ResourceQuotas</h3>
<p>A <strong><a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/">ResourceQuota</a></strong> sets hard limits on what a namespace can consume:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ResourceQuota</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">research-quota</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">hard</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requests.cpu</span><span>: </span><span style="color:#d69d85;">"20"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requests.memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">40Gi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">limits.cpu</span><span>: </span><span style="color:#d69d85;">"40"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">limits.memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">80Gi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pods</span><span>: </span><span style="color:#d69d85;">"50"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">persistentvolumeclaims</span><span>: </span><span style="color:#d69d85;">"10"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">services.loadbalancers</span><span>: </span><span style="color:#d69d85;">"2"
</span></code></pre>
<p>This ensures the genomics team can't accidentally (or intentionally) starve the clinical trials team of resources. It's the K8s equivalent of disk quotas and <code>ulimit</code> — applied at the namespace level.</p>
<h3 id="limitranges">LimitRanges</h3>
<p>While ResourceQuotas cap the <em>total</em> for a namespace, <strong><a href="https://kubernetes.io/docs/concepts/policy/limit-range/">LimitRanges</a></strong> set <em>per-pod</em> defaults and limits:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">LimitRange</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">default-limits</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">limits</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">default</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="color:#d69d85;">"500m"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">256Mi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">defaultRequest</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="color:#d69d85;">"100m"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">128Mi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">max</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="color:#d69d85;">"4"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">8Gi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">min</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="color:#d69d85;">"50m"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="background-color:#282828;color:#d69d85;">64Mi</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Container</span><span>
</span></code></pre>
<p>If a developer deploys a pod without specifying resource requests/limits, LimitRange fills in the defaults. If they request more than the max, the pod is rejected. This prevents the classic "someone deployed a pod with no limits and it ate the entire node" scenario.</p>
<p><br><br></p>
<h2 id="audit-observability">Audit & observability</h2>
<p>You can't secure what you can't see. Audit logging answers the question: <strong>who did what, when, and from where?</strong></p>
<h3 id="k8s-audit-logging">K8s audit logging</h3>
<p>The K8s API server can log every request it receives. Audit events are categorized by stage:</p>
<ul>
<li><strong>RequestReceived</strong>: The request was received but not yet processed</li>
<li><strong>ResponseStarted</strong>: Response headers sent, body not yet (for long-running requests like <code>watch</code>)</li>
<li><strong>ResponseComplete</strong>: The full response was sent</li>
<li><strong>Panic</strong>: Something went very wrong</li>
</ul>
<p>And by level:</p>
<ul>
<li><strong>None</strong>: Don't log</li>
<li><strong>Metadata</strong>: Log request metadata (user, timestamp, resource, verb) but not body</li>
<li><strong>Request</strong>: Log metadata + request body</li>
<li><strong>RequestResponse</strong>: Log everything (metadata + request body + response body)</li>
</ul>
<p>An audit policy defines what to log for different resources:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">audit.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Policy</span><span>
</span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span style="color:#608b4e;"># Log all changes to secrets at the RequestResponse level
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">level</span><span>: </span><span style="background-color:#282828;color:#d69d85;">RequestResponse</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">group</span><span>: </span><span style="color:#d69d85;">""
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>: [</span><span style="color:#d69d85;">"secrets"</span><span>]
</span><span style="color:#608b4e;"># Log pod changes at the Request level
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">level</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Request</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">group</span><span>: </span><span style="color:#d69d85;">""
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>: [</span><span style="color:#d69d85;">"pods"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">verbs</span><span>: [</span><span style="color:#d69d85;">"create"</span><span>, </span><span style="color:#d69d85;">"update"</span><span>, </span><span style="color:#d69d85;">"patch"</span><span>, </span><span style="color:#d69d85;">"delete"</span><span>]
</span><span style="color:#608b4e;"># Log everything else at Metadata level
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">level</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Metadata</span><span>
</span></code></pre>
<h3 id="eks-control-plane-logging-cloudwatch">EKS control plane logging → CloudWatch</h3>
<p>In EKS, you don't configure the audit policy directly (the control plane is managed). Instead, you enable <strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html">EKS control plane logging</a></strong> which sends logs to CloudWatch:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>aws eks update-cluster-config \
</span><span> --name my-cluster \
</span><span> --logging </span><span style="color:#d69d85;">'{
</span><span style="color:#d69d85;"> "clusterLogging": [{
</span><span style="color:#d69d85;"> "types": ["api", "audit", "authenticator", "controllerManager", "scheduler"],
</span><span style="color:#d69d85;"> "enabled": true
</span><span style="color:#d69d85;"> }]
</span><span style="color:#d69d85;"> }'
</span></code></pre>
<p>The five log types:</p>
<ul>
<li><strong>api</strong>: API server request logs</li>
<li><strong>audit</strong>: K8s audit logs (who did what)</li>
<li><strong>authenticator</strong>: IAM authentication events</li>
<li><strong>controllerManager</strong>: Controller decisions (scaling, reconciliation)</li>
<li><strong>scheduler</strong>: Pod placement decisions</li>
</ul>
<h3 id="connecting-to-cloudtrail">Connecting to CloudTrail</h3>
<p>Here's where the AWS training series comes full circle. <a href="/posts/aws-2-3/#cloudtrail-who-did-what-and-when">CloudTrail</a> records AWS API calls — <code>CreateCluster</code>, <code>UpdateNodegroup</code>, <code>AssociateEncryptionConfig</code>. CloudWatch Logs receives K8s-level audit events — <code>kubectl apply</code>, <code>kubectl delete</code>, <code>kubectl exec</code>.</p>
<p>Together, they give you the full picture:</p>
<ul>
<li><strong>CloudTrail</strong>: "Who created the cluster? Who changed the node group? Who modified IAM roles?"</li>
<li><strong>CloudWatch (K8s audit logs)</strong>: "Who deployed this pod? Who accessed this secret? Who exec'd into that container?"</li>
</ul>
<p>For a biomedical research environment handling sensitive patient data, this level of traceability isn't optional — it's often a compliance requirement (HIPAA, GDPR).</p>
<p><br><br></p>
<h2 id="best-practices-checklist">Best practices checklist</h2>
<p>Here's what "secure by default" should look like, organized by layer:</p>
<p><strong>Identity & Access</strong></p>
<ul>
<li>Apply least-privilege RBAC: namespace-scoped Roles over ClusterRoles</li>
<li>Create dedicated Service Accounts per workload; disable auto-mount for pods that don't need API access</li>
<li>Use IRSA or EKS Pod Identity for AWS access — never store AWS credentials as K8s Secrets</li>
<li>Audit RoleBindings and ClusterRoleBindings regularly</li>
</ul>
<p><strong>Network</strong></p>
<ul>
<li>Apply default-deny NetworkPolicies to every namespace</li>
<li>Segment namespaces by trust level (production vs staging, team A vs team B)</li>
<li>Use Security Groups for Pods when controlling access to AWS resources</li>
<li>Restrict API server endpoint access (private endpoint in EKS)</li>
</ul>
<p><strong>Pods</strong></p>
<ul>
<li>Enforce Pod Security Standards (at least <code>baseline</code>, ideally <code>restricted</code>)</li>
<li>Run containers as non-root, with read-only root filesystem</li>
<li>Drop all Linux capabilities, add back only what's needed</li>
<li>Use taints/tolerations to isolate sensitive workloads on dedicated nodes</li>
</ul>
<p><strong>Secrets</strong></p>
<ul>
<li>Enable etcd encryption at rest (KMS key in EKS)</li>
<li>Use an external secret store (AWS Secrets Manager, Vault) — not raw K8s Secrets</li>
<li>Rotate secrets automatically; never commit secrets to Git in plain text</li>
</ul>
<p><strong>Audit & Observability</strong></p>
<ul>
<li>Enable all EKS control plane log types (api, audit, authenticator, controllerManager, scheduler)</li>
<li>Set up alerts for privilege escalation attempts, <code>kubectl exec</code> into production pods, secret access patterns</li>
<li>Cross-reference K8s audit logs with CloudTrail for full traceability</li>
</ul>
<p><strong>Supply Chain</strong></p>
<ul>
<li>Scan container images for vulnerabilities (Trivy, Snyk, ECR image scanning)</li>
<li>Use trusted base images from verified registries</li>
<li>Sign and verify container images (cosign, Notation)</li>
<li>Pin image tags to digests (<code>image: my-app@sha256:abc123...</code> instead of <code>image: my-app:latest</code>)</li>
</ul>
<p>For a comprehensive, opinionated, and up-to-date reference, see the <a href="https://aws.github.io/aws-eks-best-practices/security/docs/">AWS EKS Best Practices Guide</a>.</p>
<br>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>Security consists of a series of layers that are stacked on top of one another. K8s gives you all the layers: RBAC, Network Policies, Pod Security Admission, audit logging, Secrets encryption. But it doesn't stack them on for you. <strong>The defaults are permissive by design</strong>, because K8s optimizes for getting things running.</p>
<p>I've got a terrible headache, I've learnt a lot of new concepts this week, but I can tell I haven't quite got the hang of them yet. In the mean time, here are resources for you to go further:</p>
<p><strong>Official documentation:</strong></p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/security/">K8s security documentation</a></li>
<li><a href="https://aws.github.io/aws-eks-best-practices/security/docs/">AWS EKS best practices guide - Security</a></li>
<li><a href="https://kubernetes.io/docs/concepts/security/pod-security-standards/">Pod Security Standards</a></li>
<li><a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">Network Policies</a></li>
<li><a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RBAC documentation</a></li>
</ul>
<p><strong>Tools:</strong></p>
<ul>
<li><a href="https://trivy.dev/">Trivy - container image scanner</a></li>
<li><a href="https://external-secrets.io/">External Secrets Operator</a></li>
<li><a href="https://sealed-secrets.netlify.app/">Sealed Secrets</a></li>
<li><a href="https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks">Calico Network Policies for EKS</a></li>
<li><a href="https://github.com/aquasecurity/kube-bench">kube-bench - CIS K8s Benchmark checker</a></li>
<li><a href="https://falco.org/">Falco - runtime security</a></li>
</ul>
<p><strong>Video tutorials:</strong></p>
<ul>
<li><a href="https://www.youtube.com/watch?v=oBf5lrmquYI">K8s security best practices</a></li>
<li><a href="https://www.youtube.com/watch?v=jvhKOAB3Jfk">RBAC explained</a></li>
<li><a href="https://www.youtube.com/watch?v=G3R24JSlGjY">Understand RBAC in k8s</a></li>
<li><a href="https://www.youtube.com/watch?v=dTKGkOiVFfA">Network Policies explained</a></li>
<li><a href="https://www.youtube.com/watch?v=HRU0aCkhgwg">7 k8s security best practices</a></li>
</ul>
<br>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/headache.jpg" alt="Too much concepts to digest" title="Too much concepts to digest"/>
AWS (Day 4)2026-02-05T00:00:00+00:002026-02-05T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-4/ <img style="display: block; margin: 0 auto; width: 600px" src="/images/albator84.webp" alt="Come on maaan, I was obliged" title="Come on maaan, I was obliged"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>My employer says I don't have the right to share the source code I've written in the course of my work. So, I'll try to say as much as I can without divulging any specific information.</p>
</li>
<li>
<p>The interactions between EKS and the broader AWS ecosystem can only be fully understood through <em>real & serious</em> practice. AWS offers a <a href="https://aws.amazon.com/free/">free tier</a> with limited resources. Start small, monitor your costs, and don't leave clusters running when you're done.</p>
</li>
<li>
<p>This article is <strong>educational content</strong>, <em>in case you missed it</em>, not a production deployment guide. Everything was intentionally simplified here for clarity. Before running EKS in production, please consult the <a href="https://aws.github.io/aws-eks-best-practices/">AWS EKS best practices guide</a> and involve your security/infrastructure teams.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece, if you catch any errors, please reach out—I'd be grateful for the feedback and happy to make updates!</p>
</li>
<li>
<p>A special thank you to <a href="https://www.linkedin.com/in/mohamed-abdoul-karim-diop-76250b127/">Abdoul Karim</a> for taking the time to re-read this article and for his honest, sharp feedback — he is the one who pointed out that it read more like a Kubernetes tutorial than an EKS article. That observation directly led to the hands-on EKS examples being added. Thanks a million to him.</p>
</li>
</ol>
<br>
<br>
<h2 id="hook">Hook</h2>
<p>This is the day 4 of the training course. The most exhausting part isn't the training itself. The most exhausting part is that I have to leave the house earlier than usual in the hope of arriving on time (as someone who usually arrive at work at 10am).</p>
<p>Today will be easier than yesterday, <em>I hope</em>, because I’m not starting from scratch; I already know a thing or two about Kubernetes. I more or less know how to write some manifests files and use them to deploy my Django projects. Let's <a href="https://www.youtube.com/watch?v=v3oK0yMZzgo">go</a>.</p>
<blockquote>
<p><em>The name Kubernetes originates from Ancient Greek: κυβερνήτης, romanized: kubernḗtēs, meaning pilot, steersman, navigator, and the etymological root of cybernetics. Kubernetes is often abbreviated as K8s, counting the eight letters between the K and the s (a numeronym).</em></p>
</blockquote>
<br>
<br>
<h2 id="table-of-contents">Table of contents</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-4/#k8s-concepts-traditional-infrastructure">K8S concepts → traditional infrastructure</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#why-containers">Why containers?</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#kubernetes-fundamentals">Kubernetes fundamentals</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#k8s-architecture">K8S architecture</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#k8s-core-objects">K8S core objects</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#k8s-configuration-tooling">K8S configuration & tooling</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#enter-eks">Enter EKS</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#eks-networking-iam-integration">EKS networking & IAM integration</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#practical-example">Practical example</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#eks-vs-alternatives">EKS vs. alternatives</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#conclusion">Conclusion</a></li>
<li><a href="https://nskm.xyz/posts/aws-4/#more-on-this-topic">More on this topic</a></li>
</ol>
<p><br><br></p>
<h2 id="k8s-concepts-traditional-infrastructure">K8S concepts → traditional infrastructure</h2>
<p>If you've been managing Linux servers and containers, you already know most of these concepts. Kubernetes just orchestrates them at scale with different names:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">K8S Concept</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Traditional Equivalent</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">What's Different?</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Cluster</strong></td>
<td style="border: 1px solid #333; padding: 10px;">A set of servers you SSH into</td>
<td style="border: 1px solid #333; padding: 10px;">Managed as a single unit, self-healing</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Control Plane</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Your Ansible/Puppet master server</td>
<td style="border: 1px solid #333; padding: 10px;">Manages scheduling, state, API; you don't run workloads on it</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Worker Node</strong></td>
<td style="border: 1px solid #333; padding: 10px;">A Linux server running your apps</td>
<td style="border: 1px solid #333; padding: 10px;">K8S schedules pods onto it automatically</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Pod</strong></td>
<td style="border: 1px solid #333; padding: 10px;">A process or <code>systemd</code> unit</td>
<td style="border: 1px solid #333; padding: 10px;">Ephemeral, gets its own IP, holds containers</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Deployment</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>systemd</code> service + rolling restart script</td>
<td style="border: 1px solid #333; padding: 10px;">Declarative desired state, automatic rollback, self-healing</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>StatefulSet</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Manually managed database instances</td>
<td style="border: 1px solid #333; padding: 10px;">Ordered startup, stable network identity, persistent storage per replica</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>DaemonSet</strong></td>
<td style="border: 1px solid #333; padding: 10px;">A service enabled on every server (<code>systemctl enable</code>)</td>
<td style="border: 1px solid #333; padding: 10px;">Ensures one pod per node, useful for log collectors, monitoring agents</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Service</strong></td>
<td style="border: 1px solid #333; padding: 10px;">DNS entry + <code>iptables</code> rules / HAProxy</td>
<td style="border: 1px solid #333; padding: 10px;">Stable endpoint for ephemeral pods, built-in load balancing</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Ingress</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Nginx or HAProxy reverse proxy config</td>
<td style="border: 1px solid #333; padding: 10px;">Declarative HTTP/HTTPS routing, TLS termination, path-based rules</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>ConfigMap</strong></td>
<td style="border: 1px solid #333; padding: 10px;">export, .env file, configuration files in /etc/</td>
<td style="border: 1px solid #333; padding: 10px;">Decoupled from the container image, injectable as env vars or mounted files</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Secret</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>gpg</code>-encrypted files, <code>pass</code></td>
<td style="border: 1px solid #333; padding: 10px;">Base64 encoded (not encrypted by default!), injectable like ConfigMaps</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Volume</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>/mnt</code>, NFS mount, LVM</td>
<td style="border: 1px solid #333; padding: 10px;">Lifecycle tied to pod (ephemeral) or independent (PersistentVolume)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Namespace</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Linux users/groups, separate directories</td>
<td style="border: 1px solid #333; padding: 10px;">Logical cluster partitioning, resource quotas, access control boundaries</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>kubectl</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>ssh</code> + <code>systemctl</code> + <code>journalctl</code></td>
<td style="border: 1px solid #333; padding: 10px;">Single CLI to manage everything in the cluster</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Helm</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>apt</code> / <code>dnf</code> package manager</td>
<td style="border: 1px solid #333; padding: 10px;">Templated K8S manifests, versioned releases, rollback support</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>kubeconfig</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>~/.ssh/config</code></td>
<td style="border: 1px solid #333; padding: 10px;">Defines which clusters you can talk to and with which credentials</td>
</tr>
</tbody>
</table>
<p><strong>The trade-off</strong>: You give up simplicity (one server, one config file, one <code>systemctl restart</code>) in exchange for automatic scaling, self-healing, and declarative infrastructure. Whether that's worth it depends on whether you're running 2 containers or 200. And if you understood everything in the table above, you can stop here, I'm serious :\</p>
<p><br><br></p>
<h2 id="want-to-know-more-okay">Want to know more ? Okay</h2>
<h3 id="why-containers">Why containers?</h3>
<p>Before Kubernetes, there were containers. Before containers, there was the classic problem: <em>"it works on my machine."</em></p>
<p>You write a Python script that processes epidemiological data. It runs perfectly on your laptop with Python 3.11, pandas 2.1, and that one obscure C library you installed six months ago and forgot about. You hand it to a colleague. It breaks. Different Python version, missing library, wrong OS. You spend half a day debugging environment issues instead of doing actual work.</p>
<p><strong>Containers <em>promise</em> to solve this.</strong> A container packages your application <em>and</em> its entire runtime environment (OS libraries, Python version, dependencies) into a single, portable image. If it runs in the container on your laptop, it runs the same way on a server in Bamako, in Luanda, or on your colleague's machine. No surprises*.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>Without containers: With containers:
</span><span>+---------------------+ +-----------------------+
</span><span>| App A (Python 3.9) | | +-------------------+ |
</span><span>| App B (Python 3.11) | | | Container A | |
</span><span>| App C (Java 17) | | | Python 3.9 + App | |
</span><span>| Conflicting deps! | | +-------------------+ |
</span><span>| Shared OS libraries | | +-------------------+ |
</span><span>| "Who installed | | | Container B | |
</span><span>| what?" | | | Python 3.11 + App | |
</span><span>| | | +-------------------+ |
</span><span>| One bare-metal | | +-------------------+ |
</span><span>| server | | | Container C | |
</span><span>| | | | Java 17 + App | |
</span><span>+---------------------+ | +-------------------+ |
</span><span> | Isolated, no |
</span><span> | conflicts |
</span><span> +-----------------------+
</span></code></pre>
<p><strong>So why do you need Kubernetes?</strong></p>
<p>Containers solve the packaging problem. But in production, new questions appear:</p>
<ul>
<li>You have 15 containers across 6 servers, how do you know which server has capacity for a new one?</li>
<li>A server crashes at 2am, who restarts the containers that were running on it?</li>
<li>Traffic spikes on Monday morning, how do you spin up more copies of your web app?</li>
<li>You push a bad update, how do you roll back quickly?</li>
<li>Your containers need to talk to each other, who manages the networking and service discovery?</li>
</ul>
<p>You <em>could</em> solve all of this with bash scripts, cron jobs, and Ansible playbooks. People did, for years. It worked, until it didn't — usually at the worst possible moment.</p>
<p><strong><a href="https://kubernetes.io/">Kubernetes (K8S)</a></strong> is an open-source platform created at Google by <a href="https://www.linkedin.com/in/jbeda">Joe Beda</a>, <a href="https://www.linkedin.com/in/brendan-burns-487aa590">Brendan Burns</a>, and <a href="https://www.linkedin.com/in/craigmcluckie">Craig McLuckie</a>. Announced in <a href="https://cloud.google.com/blog/products/containers-kubernetes/from-google-to-the-world-the-kubernetes-origin-story">June 2014</a>, it grew out of Google's internal cluster manager <a href="https://research.google/pubs/large-scale-cluster-management-at-google-with-borg/">Borg</a>, which had been orchestrating containers at massive scale for over a decade. Inspired by Docker's rise, the three founders saw the need for something that could orchestrate <em>many</em> containers across <em>many</em> machines. Google <a href="https://www.cncf.io/reports/kubernetes-project-journey-report/">donated Kubernetes to the CNCF</a> in 2015, and it quickly became the industry standard.</p>
<p>You tell K8S <em>what</em> you want (3 replicas of my app, exposed on port 443, with 2GB of RAM each), and it figures out <em>how</em> to make it happen. If something breaks, K8S fixes it automatically. <em>That's the pitch</em>.</p>
<hr />
<h3 id="kubernetes-fundamentals">Kubernetes fundamentals</h3>
<p>Kubernetes operates on one core principle: <strong>declarative configuration</strong>. Instead of telling the system <em>how</em> to do things step by step (imperative), you describe <em>what</em> you want the end state to look like, and K8s figures out how to get there.</p>
<p>On a traditional Linux server, you might do:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># Imperative: you tell the system every step
</span><span>podman run -d --name webapp -p 8080:80 my-app:v2
</span><span>podman stop webapp-old
</span><span>podman rm webapp-old
</span></code></pre>
<p>With Kubernetes, you write a manifest:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># Declarative: you describe the desired state
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Deployment</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">replicas</span><span>: </span><span style="color:#b5cea8;">3
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-app:v2</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">containerPort</span><span>: </span><span style="color:#b5cea8;">80
</span></code></pre>
<p>You apply it (<code>kubectl apply -f webapp.yaml</code>), and K8s handles the rest: scheduling pods on nodes with available resources, rolling out the new version, terminating old pods, restarting anything that crashes. You described <em>what</em> you want — 3 replicas of <code>my-app:v2</code> on port 80 — and K8S continuously works to make reality match your description. This is called the <strong><a href="https://kubernetes.io/docs/concepts/architecture/controller/">reconciliation loop</a></strong>: K8s constantly compares the desired state (your manifest) with the actual state (what's running), and corrects any drift.</p>
<p><strong>Nota Bene:</strong> Every K8s object — Pods, Services, Deployments — is just a piece of desired state that a controller is responsible for reconciling.</p>
<h3 id="k8s-architecture">K8s architecture</h3>
<p>A Kubernetes cluster is split into two layers: the <strong>control plane</strong> that makes decisions, and the <strong>worker nodes</strong> that run your actual workloads.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span> +--------------------------------------------------------------+
</span><span> | K8S CLUSTER |
</span><span> | |
</span><span> | +-----------------------------+ |
</span><span> | | CONTROL PLANE | |
</span><span> | | | |
</span><span> | | +----------+ +---------+ | |
</span><span> | | |Scheduler | |API | |<-- kubectl talks here |
</span><span> | | | | |Server | | |
</span><span> | | +----------+ +---------+ | |
</span><span> | | +----------+ +---------+ | |
</span><span> | | |Controller| | etcd | |<-- cluster state lives |
</span><span> | | |Manager | | (kv) | | here |
</span><span> | | +----------+ +---------+ | |
</span><span> | +-----------------------------+ |
</span><span> | | ^ |
</span><span> | instructs | | reports back |
</span><span> | v | |
</span><span> | |
</span><span> | +-------------+ +-------------+ +-------------+ |
</span><span> | | WORKER 1 | | WORKER 2 | | WORKER 3 | |
</span><span> | | | | | | | |
</span><span> | | +---------+ | | +---------+ | | +---------+ | |
</span><span> | | |kubelet | | | |kubelet | | | |kubelet | | |
</span><span> | | +---------+ | | +---------+ | | +---------+ | |
</span><span> | | |kube | | | |kube | | | |kube | | |
</span><span> | | |proxy | | | |proxy | | | |proxy | | |
</span><span> | | +---------+ | | +---------+ | | +---------+ | |
</span><span> | | |container| | | |container| | | |container| | |
</span><span> | | |runtime | | | |runtime | | | |runtime | | |
</span><span> | | +---------+ | | +---------+ | | +---------+ | |
</span><span> | | |Pod A | | | |Pod C | | | |Pod E | | |
</span><span> | | |Pod B | | | |Pod D | | | | | | |
</span><span> | | +---------+ | | +---------+ | | +---------+ | |
</span><span> | +-------------+ +-------------+ +-------------+ |
</span><span> +--------------------------------------------------------------+
</span></code></pre>
<p><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#control-plane-components">Control Plane</a></strong> (also called master nodes) — the brain of the cluster. It doesn't run your applications; it manages everything else:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Component</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Role</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Traditional Equivalent</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#kube-apiserver">API Server</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Front door for all cluster operations. Every <code>kubectl</code> command, every internal component, talks through it</td>
<td style="border: 1px solid #333; padding: 10px;">The SSH daemon + a REST API on your server</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#etcd">etcd</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Distributed key-value store holding the entire cluster state</td>
<td style="border: 1px solid #333; padding: 10px;">Your <code>/etc</code> directory + a database, replicated</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#kube-scheduler">Scheduler</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Decides which worker node should run a new pod based on resource availability and constraints</td>
<td style="border: 1px solid #333; padding: 10px;">You, manually picking which server to deploy to</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager">Controller Manager</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Runs the reconciliation loops — watches desired state vs actual state and corrects drift</td>
<td style="border: 1px solid #333; padding: 10px;">Your Ansible playbooks running on a schedule</td>
</tr>
</tbody>
</table>
<p>The control plane requires fewer resources than worker nodes but is <em>far more critical</em>. In production, you run at least 2 control plane nodes (one primary, one for recovery) so the cluster survives a node failure.</p>
<br>
<p><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#node-components">Worker Nodes</a></strong> are where the heavy lifting happens: we are talking about bigger machines, more CPU, more RAM, hosting all your application pods. They're also more expendable than control plane nodes, if a worker dies, the scheduler simply reschedules its pods onto surviving nodes. Each worker node has three main components:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Component</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Description</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#kubelet">Kubelet</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">An agent running on each node. It receives pod specifications from the API Server, ensures the containers described in those specs are running and healthy, and reports status back. Think of it as a local <code>systemd</code> that takes orders from the control plane.</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/overview/components/#kube-proxy">Kube-proxy</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">A network component on each node that maintains network rules, enabling communication to and from your pods. It's what makes Services work — routing traffic to the right pod regardless of which node it's on.</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/">Container runtime</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">The software that actually runs containers. K8s supports <a href="https://containerd.io/">containerd</a>, <a href="https://cri-o.io/">CRI-O</a>, and any runtime implementing the <a href="https://kubernetes.io/docs/concepts/architecture/cri/">Container Runtime Interface (CRI)</a>. Docker was the original default but was <a href="https://kubernetes.io/blog/2022/02/17/dockershim-faq/">deprecated as a runtime in K8s 1.24</a>.</td>
</tr>
</tbody>
</table>
<h3 id="k8s-core-objects">K8s core objects</h3>
<p>Now that you understand the architecture, let's walk through the objects you'll actually work with \o/</p>
<h4 id="pods">Pods</h4>
<p>A <strong><a href="https://kubernetes.io/docs/concepts/workloads/pods/">Pod</a></strong> is the smallest deployable unit in Kubernetes. It's an abstraction over one or more containers, usually running a single application. Each pod gets its own IP address and can hold sidecar containers (supporting processes like log shippers or proxies).</p>
<p><a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/assembly_working-with-pods_building-running-and-managing-containers">Pods</a> are <strong>ephemeral</strong> — they're designed to be disposable. If a pod dies, it's gone. K8s creates a new one to replace it, with a different IP. This is why you never talk to pods directly in production.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Pod</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-app</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">nginx:1.25</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">containerPort</span><span>: </span><span style="color:#b5cea8;">80
</span></code></pre>
<p>On a traditional server, a pod is roughly equivalent to a running process or a <code>systemd</code> unit. The difference? K8s manages the pod's lifecycle automatically.</p>
<h4 id="services">Services</h4>
<p>Since pods are ephemeral, you need a stable endpoint to reach them. That's what a <strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/">Service</a></strong> provides — a permanent IP address and DNS name that load-balances across a set of pods. If a pod dies and gets replaced, the Service keeps working.</p>
<p><strong>Types of Services:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Type</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Scope</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Use Case</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip">ClusterIP</a></strong> (default)</td>
<td style="border: 1px solid #333; padding: 10px;">Internal cluster IP</td>
<td style="border: 1px solid #333; padding: 10px;">Backend services that don't need external access</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport">NodePort</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Exposes on each node's IP at a static port</td>
<td style="border: 1px solid #333; padding: 10px;">Quick external access for testing (not recommended for production)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer">LoadBalancer</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Cloud provider's load balancer</td>
<td style="border: 1px solid #333; padding: 10px;">Production external access (on AWS, creates an ELB/ALB)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/#externalname">ExternalName</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">DNS CNAME mapping</td>
<td style="border: 1px solid #333; padding: 10px;">Pointing to external services (<code>database.example.com</code>)</td>
</tr>
</tbody>
</table>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Service</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp-service</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span> </span><span style="color:#608b4e;"># Matches pods with this label
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="color:#b5cea8;">80
</span><span> </span><span style="background-color:#282828;color:#569cd6;">targetPort</span><span>: </span><span style="color:#b5cea8;">8080
</span><span> </span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">LoadBalancer</span><span>
</span></code></pre>
<p>Think of a Service as your <code>iptables</code> rules + HAProxy + DNS entry, all managed declaratively.</p>
<h4 id="ingress">Ingress</h4>
<p>A <strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/">Service</a></strong> can expose your app, but what if you have 10 services and want them all behind a single domain with path-based routing? That's what <strong><a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a></strong> does — HTTP/HTTPS routing with TLS termination and name-based virtual hosting.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">networking.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Ingress</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-ingress</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">host</span><span>: </span><span style="background-color:#282828;color:#d69d85;">example.com</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">http</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">paths</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/app</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pathType</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Prefix</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">backend</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp-service</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">number</span><span>: </span><span style="color:#b5cea8;">80
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/api</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pathType</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Prefix</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">backend</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">api-service</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">number</span><span>: </span><span style="color:#b5cea8;">8080
</span></code></pre>
<p>An <strong><a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/">Ingress Controller</a></strong> (like <a href="https://kubernetes.github.io/ingress-nginx/">Nginx Ingress Controller</a> or AWS's <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/">ALB Ingress Controller</a>) is required to implement the rules. On a single Linux server, this is just Nginx or HAProxy configured as a reverse proxy.</p>
<h4 id="configmaps">ConfigMaps</h4>
<p>A <strong><a href="https://kubernetes.io/docs/concepts/configuration/configmap/">ConfigMap</a></strong> stores configuration data separately from your container image. You can inject it as environment variables or mount it as files in a pod. This decouples config from code — no rebuilding images just to change a database URL.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ConfigMap</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">app-config</span><span>
</span><span style="background-color:#282828;color:#569cd6;">data</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">DATABASE_URL</span><span>: </span><span style="color:#d69d85;">"postgres://db.example.com:5432/mydb"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">LOG_LEVEL</span><span>: </span><span style="color:#d69d85;">"info"
</span></code></pre>
<p>Reference it in a pod:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-app:v1</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">envFrom</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">configMapRef</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">app-config</span><span>
</span></code></pre>
<p>On a traditional server, this is just files in <code>/etc/myapp/</code>. K8s makes them version-controlled and injectable.</p>
<h4 id="secrets">Secrets</h4>
<p><strong><a href="https://kubernetes.io/docs/concepts/configuration/secret/">Secrets</a></strong> work exactly like ConfigMaps, but they're intended for sensitive data (passwords, API keys). They're stored base64-encoded, <strong>not encrypted by default</strong> in etcd. For real encryption at rest, enable <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/">etcd encryption</a> or use external secret managers like <a href="https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html">AWS Secrets Manager</a> with <a href="https://external-secrets.io/">External Secrets Operator</a>.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Secret</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">db-credentials</span><span>
</span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Opaque</span><span>
</span><span style="background-color:#282828;color:#569cd6;">data</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">username</span><span>: </span><span style="background-color:#282828;color:#d69d85;">YWRtaW4=</span><span> </span><span style="color:#608b4e;"># base64("admin")
</span><span> </span><span style="background-color:#282828;color:#569cd6;">password</span><span>: </span><span style="background-color:#282828;color:#d69d85;">cGFzc3dvcmQ=</span><span> </span><span style="color:#608b4e;"># base64("password")
</span></code></pre>
<p>On a Linux server, this is like files encrypted with <code>gpg</code> or managed by <code>pass</code>. The K8s equivalent just integrates better with the pod lifecycle.</p>
<h4 id="volumes">Volumes</h4>
<p>Containers are stateless by default — data disappears when they restart. <strong><a href="https://kubernetes.io/docs/concepts/storage/volumes/">Volumes</a></strong> attach storage to pods so data persists. K8s supports many volume types: local disks, NFS, cloud provider block storage (AWS EBS, Azure Disk), and more.</p>
<p>For truly persistent storage that outlives a pod, use a <strong><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">PersistentVolume (PV)</a></strong> and <strong><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims">PersistentVolumeClaim (PVC)</a></strong>.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">PersistentVolumeClaim</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data-pvc</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">accessModes</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#d69d85;">ReadWriteOnce</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requests</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">storage</span><span>: </span><span style="background-color:#282828;color:#d69d85;">10Gi</span><span>
</span><span>---
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Pod</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">db-pod</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres:16</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumeMounts</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">mountPath</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/var/lib/postgresql/data</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data-volume</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumes</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data-volume</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">persistentVolumeClaim</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">claimName</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data-pvc</span><span>
</span></code></pre>
<p>On a single Linux machine, this is just mounting <code>/mnt/data</code> or an NFS share. K8s abstracts it so you can move workloads across nodes without reconfiguring mount points.</p>
<h4 id="deployments">Deployments</h4>
<p>You rarely create pods directly. Instead, you use a <strong><a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a></strong> — a blueprint for stateless applications. It manages scaling, replica counts, rolling updates, and rollbacks.</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Deployment</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">replicas</span><span>: </span><span style="color:#b5cea8;">3
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-app:v2</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">containerPort</span><span>: </span><span style="color:#b5cea8;">8080
</span></code></pre>
<p>Common operations:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl apply -f deployment.yaml </span><span style="color:#608b4e;"># Create or update
</span><span>kubectl scale deployment webapp --replicas=5 </span><span style="color:#608b4e;"># Scale to 5 pods
</span><span>kubectl set image deployment/webapp webapp=my-app:v3 </span><span style="color:#608b4e;"># Rolling update
</span><span>kubectl rollout undo deployment/webapp </span><span style="color:#608b4e;"># Rollback to previous version
</span><span>kubectl rollout status deployment/webapp </span><span style="color:#608b4e;"># Watch rollout progress
</span></code></pre>
<p>On a traditional server, this is <code>systemd</code> units + a rolling restart script managed by <a href="https://docs.ansible.com/">Ansible</a>. K8s does it declaratively, with automatic health checks and rollback on failure.</p>
<h4 id="statefulsets">StatefulSets</h4>
<p><strong><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSets</a></strong> are for stateful applications like databases. Unlike Deployments, they provide:</p>
<ul>
<li><strong>Stable network identities</strong> (pod names like <code>db-0</code>, <code>db-1</code>, <code>db-2</code>)</li>
<li><strong>Ordered startup and shutdown</strong> (db-0 starts before db-1)</li>
<li><strong>Persistent storage per pod</strong> (each pod gets its own PVC)</li>
</ul>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">StatefulSet</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">serviceName</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">replicas</span><span>: </span><span style="color:#b5cea8;">3
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">postgres:16</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumeMounts</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">mountPath</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/var/lib/postgresql/data</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumeClaimTemplates</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">data</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">accessModes</span><span>: [</span><span style="color:#d69d85;">"ReadWriteOnce"</span><span>]
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requests</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">storage</span><span>: </span><span style="background-color:#282828;color:#d69d85;">20Gi</span><span>
</span></code></pre>
<p>From my online research, many teams prefer to run databases <em>outside</em> K8s (RDS, managed PostgreSQL) and only use K8s for stateless workloads. I guess nobody wants to resolve an issue requiring expertise in Postgresql AND Kubernetes layers. <em>If you run your database inside k8s using a StatefulSet, please, let me know what are the good, the bad & the ugly.</em>..... { °_°} <em>Someone's whispering in my ear that I should take a look at <a href="https://cloudnative-pg.io/">CloudnativePG</a>.</em></p>
<h4 id="daemonsets">DaemonSets</h4>
<p>A <strong><a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">DaemonSet</a></strong> ensures a copy of a pod runs on every node (or selected nodes). Useful for cluster-wide services like: log collectors (Fluentd, Filebeat), monitoring agents (Prometheus Node Exporter, Datadog agent), network plugins (Calico, Cilium).</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">DaemonSet</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">node-exporter</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">node-exporter</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">node-exporter</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">node-exporter</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">prom/node-exporter:latest</span><span>
</span></code></pre>
<p>On a traditional infrastructure, this is like running <code>systemctl enable monitoring-agent</code> on every server. K8s does it automatically, including on new nodes as they join the cluster.</p>
<h3 id="k8s-configuration-tooling">K8s configuration & tooling</h3>
<p>Now that we know the objects, how do we actually work with them?</p>
<h4 id="kubectl-the-k8s-cli">kubectl — the K8s CLI</h4>
<p><strong><a href="https://kubernetes.io/docs/reference/kubectl/">kubectl</a></strong> is the official command-line interface (CLI) to interact with and manage Kubernetes clusters. It's the equivalent of <code>ssh</code> + <code>systemctl</code> + <code>podman</code> all rolled into one.</p>
<p>Common commands:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># View resources
</span><span>kubectl get pods </span><span style="color:#608b4e;"># List all pods in current namespace
</span><span>kubectl get pods -n kube-system </span><span style="color:#608b4e;"># List pods in kube-system namespace
</span><span>kubectl get pods -A </span><span style="color:#608b4e;"># List pods across all namespaces
</span><span>kubectl get deployments,services,ingress </span><span style="color:#608b4e;"># Multiple resource types
</span><span>
</span><span style="color:#608b4e;"># Describe details
</span><span>kubectl describe pod my-pod </span><span style="color:#608b4e;"># Full details, events, status
</span><span>kubectl logs my-pod </span><span style="color:#608b4e;"># View logs
</span><span>kubectl logs my-pod -f </span><span style="color:#608b4e;"># Follow logs (like tail -f)
</span><span>kubectl logs my-pod --previous </span><span style="color:#608b4e;"># Logs from crashed container
</span><span>
</span><span style="color:#608b4e;"># Apply manifests
</span><span>kubectl apply -f deployment.yaml </span><span style="color:#608b4e;"># Create/update from file
</span><span>kubectl apply -f ./manifests/ </span><span style="color:#608b4e;"># Apply all YAML in directory
</span><span>kubectl delete -f deployment.yaml </span><span style="color:#608b4e;"># Delete resources
</span><span>
</span><span style="color:#608b4e;"># Direct manipulation (less common, prefer apply)
</span><span>kubectl scale deployment webapp --replicas=5
</span><span>kubectl set image deployment/webapp webapp=my-app:v3
</span><span>
</span><span style="color:#608b4e;"># Debugging
</span><span>kubectl exec -it my-pod</span><span style="color:#569cd6;"> --</span><span> /bin/bash </span><span style="color:#608b4e;"># Shell into running pod
</span><span>kubectl port-forward pod/my-pod 8080:80 </span><span style="color:#608b4e;"># Forward local port to pod
</span><span>kubectl top nodes </span><span style="color:#608b4e;"># Resource usage per node
</span><span>kubectl top pods </span><span style="color:#608b4e;"># Resource usage per pod
</span></code></pre>
<h4 id="kubeconfig-cluster-credentials">kubeconfig — cluster credentials</h4>
<p>Your cluster credentials live in the <strong><a href="https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/">kubeconfig file</a></strong>, usually at <code>~/.kube/config</code>. It defines:</p>
<ul>
<li><strong>Clusters</strong>: API server endpoints and certificates</li>
<li><strong>Users</strong>: Authentication credentials (certs, tokens, OIDC)</li>
<li><strong>Contexts</strong>: A pairing of cluster + user + namespace</li>
</ul>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Config</span><span>
</span><span style="background-color:#282828;color:#569cd6;">clusters</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">cluster</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">server</span><span>: </span><span style="background-color:#282828;color:#d69d85;">https://eks-cluster.eu-west-3.eks.amazonaws.com</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">production</span><span>
</span><span style="background-color:#282828;color:#569cd6;">contexts</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">context</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cluster</span><span>: </span><span style="background-color:#282828;color:#d69d85;">production</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">user</span><span>: </span><span style="background-color:#282828;color:#d69d85;">admin</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">default</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">prod-context</span><span>
</span><span style="background-color:#282828;color:#569cd6;">current-context</span><span>: </span><span style="background-color:#282828;color:#d69d85;">prod-context</span><span>
</span><span style="background-color:#282828;color:#569cd6;">users</span><span>:
</span><span>- </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">admin</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">user</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">token</span><span>: </span><span style="background-color:#282828;color:#d69d85;">eyJhbGciOiJSUzI1NiIsImtpZCI6...</span><span>
</span></code></pre>
<p>This is the equivalent of <code>~/.ssh/config</code> for SSH connections — defining which clusters you can access and how to authenticate.</p>
<h4 id="manifest-files-infrastructure-as-code">Manifest files — infrastructure as code</h4>
<p>All K8s resources are defined in <strong>manifest files</strong> — YAML or JSON documents. You store them in version control alongside your application code, making infrastructure changes auditable and reviewable.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>my-app/
</span><span>├── deployment.yaml
</span><span>├── service.yaml
</span><span>├── ingress.yaml
</span><span>├── configmap.yaml
</span><span>└── secrets.yaml # (encrypted in git using tools like git-crypt or sealed-secrets)
</span></code></pre>
<p>Apply them all at once:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl apply -f ./k8s/
</span></code></pre>
<p>Managed K8s services (<a href="https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html">EKS</a>, <a href="https://docs.cloud.google.com/kubernetes-engine/docs">GKE</a>) also let you manage some resources through their web consoles or CLIs. If you ask me, I'll tell you the best practice is <strong>GitOps</strong>: all configuration in <a href="https://git-scm.com/docs">Git</a>, applied via CI/CD or tools like <a href="https://argoproj.github.io/cd/">ArgoCD</a> or <a href="https://fluxcd.io/">Flux</a>.</p>
<h4 id="helm-the-k8s-package-manager">Helm — the K8s package manager</h4>
<p>Writing YAML manifests for every environment (dev, staging, prod) gets repetitive. You need to change image tags, replica counts, resource limits — but the structure stays the same. <strong><a href="https://helm.sh/">Helm</a></strong> solves this by templating manifests. I need to master this part.</p>
<p>A <strong>Helm chart</strong> is a collection of templated YAML files plus a <code>values.yaml</code> file for customization:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># values.yaml
</span><span style="background-color:#282828;color:#569cd6;">replicaCount</span><span>: </span><span style="color:#b5cea8;">3
</span><span style="background-color:#282828;color:#569cd6;">image</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">repository</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-app</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">tag</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v2.0</span><span>
</span><span style="background-color:#282828;color:#569cd6;">service</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="color:#b5cea8;">80
</span></code></pre>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># templates/deployment.yaml
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Deployment</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">.Chart.Name</span><span> }}
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">replicas</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">.Values.replicaCount</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: {{ </span><span style="background-color:#282828;color:#d69d85;">.Chart.Name</span><span> }}
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="color:#d69d85;">"{{ .Values.image.repository }}:{{ .Values.image.tag }}"
</span></code></pre>
<p>Install a chart:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>helm install my-app ./my-chart
</span><span>helm install my-app ./my-chart --set replicaCount=5 </span><span style="color:#608b4e;"># Override values
</span><span>helm upgrade my-app ./my-chart </span><span style="color:#608b4e;"># Update
</span><span>helm rollback my-app </span><span style="color:#608b4e;"># Rollback
</span><span>helm uninstall my-app </span><span style="color:#608b4e;"># Remove everything
</span></code></pre>
<p>Helm also has a massive <a href="https://artifacthub.io/">public chart repository</a> — want to deploy PostgreSQL, Redis, Nginx Ingress Controller? There's a chart for that.</p>
<p>Think of Helm as the <code>apt</code> or <code>dnf</code> of Kubernetes.</p>
<h3 id="enter-eks">Enter EKS</h3>
<p>Everything up to now has been generic Kubernetes — you could run it on bare metal, in your basement, or in any cloud. Now let's talk about <strong><a href="https://aws.amazon.com/eks/">Amazon EKS (Elastic Kubernetes Service)</a></strong>, AWS's managed Kubernetes offering.</p>
<p>The main alternatives to EKS for managing K8s include options coming from the <em><a href="#">Hyperscalers</a> (yeah, Hyperscalers, I like the name)</em> such as Google Kubernetes Engine (GKE) (often considered to be more powerful), Azure Kubernetes Service (AKS) (ideal for the Micros10p ecosystem). For hybrid or on-premises management, Rancher and Red Hat OpenShift are robust alternatives. As <em>a proud</em> DigitalOcean user, I should mention <a href="https://www.digitalocean.com/products/kubernetes">DigitalOcean Kubernetes (DOKS)</a>.</p>
<p><strong>What EKS manages for you:</strong></p>
<p>With self-managed Kubernetes, <em>you</em> install, patch, upgrade, and monitor the API server, etcd, scheduler, and controller manager. If etcd crashes at 3am, someone will be paged. With EKS, AWS runs the control plane for you:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">You Manage (Self-hosted K8s)</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">AWS Manages (EKS)</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Control plane nodes (HA setup, patching, upgrades)</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✓</strong> Fully managed, multi-AZ by default</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">etcd backups and disaster recovery</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✓</strong> Automated backups</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Control plane scaling</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✓</strong> Auto-scales based on cluster size</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">API server availability</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✓</strong> 99.95% SLA</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Security patches for control plane</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✓</strong> AWS handles it</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Worker nodes (EC2 instances)</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✗</strong> You still manage these</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Application deployments</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✗</strong> Still your responsibility</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Cluster monitoring and logging</td>
<td style="border: 1px solid #333; padding: 10px;"><strong>✗</strong> You configure CloudWatch or Prometheus</td>
</tr>
</tbody>
</table>
<p>EKS is a <strong>managed control plane</strong>, not a fully managed Kubernetes. You still provision and manage worker nodes (EC2 instances), configure networking <a href="/posts/aws-1/#network">VPCs</a>, and set up <a href="/posts/aws-2-3/#iam-policies-how-aws-actually-decides">IAM roles</a>, etc...</p>
<p><strong>Creating an EKS cluster with eksctl:</strong></p>
<p><strong><a href="https://eksctl.io/">eksctl</a></strong> is the official CLI for EKS. It abstracts away the complexity of creating, managing, and operating Amazon Elastic Kubernetes Service (Amazon EKS) clusters. Written in Go, eksctl provides a declarative syntax through YAML configurations and CLI commands to handle complex EKS cluster operations that would otherwise require multiple manual steps across different AWS services.</p>
<p>Simple cluster creation using the <a href="https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html">cli</a> (assuming you have the right sets or roles to do so):</p>
<p>In plain english:</p>
<blockquote>
<p>"Create a new Kubernetes cluster named my-cluster in the Paris region (eu-west-3). Add a group of worker nodes called standard-workers, each being a t3.medium machine. Start with 3 nodes running, but allow the cluster to scale down to a minimum of 1 node and up to a maximum of 4 nodes if needed. Make this node group managed by AWS (meaning AWS handles patching and updates for me)."</p>
</blockquote>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>eksctl create cluster \
</span><span> --name my-cluster \
</span><span> --region eu-west-3 \
</span><span> --nodegroup-name standard-workers \
</span><span> --node-type t3.medium \
</span><span> --nodes 3 \
</span><span> --nodes-min 1 \
</span><span> --nodes-max 4 \
</span><span> --managed
</span></code></pre>
<p>This command:</p>
<ol>
<li>Creates a new VPC with 8 <a href="https://docs.aws.amazon.com/eks/latest/eksctl/vpc-configuration.html">subnets</a></li>
<li>Deploys the EKS control plane</li>
<li>Launches a managed node group (3 t3.medium EC2 instances)</li>
<li>Configures Auto Scaling (1-4 nodes)</li>
<li>Sets up kubectl access automatically</li>
</ol>
<p>After 10-15 minutes, your cluster is ready:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl get nodes
</span><span style="color:#608b4e;"># NAME STATUS ROLES AGE
</span><span style="color:#608b4e;"># ip-192-168-1-10.eu-west-3.compute.internal Ready <none> 2m
</span><span style="color:#608b4e;"># ip-192-168-2-20.eu-west-3.compute.internal Ready <none> 2m
</span><span style="color:#608b4e;"># ip-192-168-3-30.eu-west-3.compute.internal Ready <none> 2m
</span></code></pre>
<p><strong>Advanced configuration</strong> (using a config file):</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># cluster.yaml
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">eksctl.io/v1alpha5</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ClusterConfig</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">production-cluster</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">region</span><span>: </span><span style="background-color:#282828;color:#d69d85;">eu-west-3</span><span>
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">vpc</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">id</span><span>: </span><span style="background-color:#282828;color:#d69d85;">vpc-0123456789abcdef</span><span> </span><span style="color:#608b4e;"># Use existing VPC from aws1
</span><span> </span><span style="background-color:#282828;color:#569cd6;">subnets</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">private</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">eu-west-3a</span><span>: { </span><span style="background-color:#282828;color:#569cd6;">id</span><span>: </span><span style="background-color:#282828;color:#d69d85;">subnet-private-a</span><span> }
</span><span> </span><span style="background-color:#282828;color:#569cd6;">eu-west-3b</span><span>: { </span><span style="background-color:#282828;color:#569cd6;">id</span><span>: </span><span style="background-color:#282828;color:#d69d85;">subnet-private-b</span><span> }
</span><span> </span><span style="background-color:#282828;color:#569cd6;">eu-west-3c</span><span>: { </span><span style="background-color:#282828;color:#569cd6;">id</span><span>: </span><span style="background-color:#282828;color:#d69d85;">subnet-private-c</span><span> }
</span><span>
</span><span style="background-color:#282828;color:#569cd6;">managedNodeGroups</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">general-purpose</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">instanceType</span><span>: </span><span style="background-color:#282828;color:#d69d85;">t3.large</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">minSize</span><span>: </span><span style="color:#b5cea8;">2
</span><span> </span><span style="background-color:#282828;color:#569cd6;">maxSize</span><span>: </span><span style="color:#b5cea8;">10
</span><span> </span><span style="background-color:#282828;color:#569cd6;">desiredCapacity</span><span>: </span><span style="color:#b5cea8;">3
</span><span> </span><span style="background-color:#282828;color:#569cd6;">volumeSize</span><span>: </span><span style="color:#b5cea8;">50
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ssh</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">allow</span><span>: </span><span style="color:#569cd6;">true
</span><span> </span><span style="background-color:#282828;color:#569cd6;">publicKeyName</span><span>: </span><span style="background-color:#282828;color:#d69d85;">my-keypair</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">workload</span><span>: </span><span style="background-color:#282828;color:#d69d85;">general</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">tags</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">team</span><span>: </span><span style="background-color:#282828;color:#d69d85;">platform</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">environment</span><span>: </span><span style="background-color:#282828;color:#d69d85;">production</span><span>
</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">compute-optimized</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">instanceType</span><span>: </span><span style="background-color:#282828;color:#d69d85;">c5.2xlarge</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">minSize</span><span>: </span><span style="color:#b5cea8;">0
</span><span> </span><span style="background-color:#282828;color:#569cd6;">maxSize</span><span>: </span><span style="color:#b5cea8;">5
</span><span> </span><span style="background-color:#282828;color:#569cd6;">desiredCapacity</span><span>: </span><span style="color:#b5cea8;">1
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">workload</span><span>: </span><span style="background-color:#282828;color:#d69d85;">compute-intensive</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">taints</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">key</span><span>: </span><span style="background-color:#282828;color:#d69d85;">compute-intensive</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">value</span><span>: </span><span style="color:#d69d85;">"true"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">effect</span><span>: </span><span style="background-color:#282828;color:#d69d85;">NoSchedule</span><span>
</span></code></pre>
<p>Apply it:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>eksctl create cluster -f cluster.yaml
</span></code></pre>
<p><strong>EKS vs. self-managed Kubernetes:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Factor</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">EKS</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Self-Managed on EC2</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Setup time</strong></td>
<td style="border: 1px solid #333; padding: 10px;">15 minutes (eksctl)</td>
<td style="border: 1px solid #333; padding: 10px;">Hours to days (kubeadm, Terraform, Ansible)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Control plane HA</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Built-in, multi-AZ</td>
<td style="border: 1px solid #333; padding: 10px;">You configure and maintain</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Upgrades</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>eksctl upgrade cluster</code></td>
<td style="border: 1px solid #333; padding: 10px;">Manual, risky, time-consuming</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Cost</strong></td>
<td style="border: 1px solid #333; padding: 10px;">$73/month + nodes</td>
<td style="border: 1px solid #333; padding: 10px;">Just node costs, but more ops time</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>AWS integration</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Native (IAM, VPC, ELB, EBS)</td>
<td style="border: 1px solid #333; padding: 10px;">Manual integration required</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Flexibility</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Limited control plane customization</td>
<td style="border: 1px solid #333; padding: 10px;">Full control</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>When to use</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Most production workloads</td>
<td style="border: 1px solid #333; padding: 10px;">Cost-sensitive, control plane customization needed</td>
</tr>
</tbody>
</table>
<p>For most teams, EKS is worth the €€€/month. You're paying AWS to handle the hard parts (HA, backups, upgrades, security patches) so you can focus on running applications, not babysitting control planes.</p>
<h3 id="eks-networking-iam-integration">EKS networking & IAM integration</h3>
<p>EKS isn't just Kubernetes on AWS — it's deeply integrated with AWS services. Three key integrations make EKS feel native to the AWS ecosystem.</p>
<h4 id="1-vpc-cni-pods-get-real-vpc-ips">1. VPC CNI — Pods get real VPC IPs</h4>
<p>In standard Kubernetes, pods get IP addresses from an internal overlay network (like Calico or Flannel). Pods can talk to each other, but they're isolated from the rest of your infrastructure. If you want a pod to access an RDS database in your VPC, you need to configure routing.</p>
<p>EKS uses the <strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html">AWS VPC CNI plugin</a></strong> instead. Every pod gets an IP address <em>directly from your VPC subnets</em> — the same subnets your EC2 instances and RDS databases live in. This means:</p>
<ul>
<li><strong>Pods are first-class VPC citizens</strong>: They appear in VPC Flow Logs, Security Groups apply to them, NACLs filter their traffic</li>
<li><strong>No NAT for pod-to-pod traffic</strong>: Pods communicate directly via VPC routing</li>
<li><strong>Simplified networking</strong>: Your pod at <code>10.0.1.50</code> can directly connect to your RDS instance at <code>10.0.2.100</code> — no special configuration needed</li>
</ul>
<p><strong>The trade-off</strong>: Each worker node has a limited number of <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html">Elastic Network Interfaces (ENIs)</a>, and each ENI has a limited number of secondary IP addresses. This caps how many pods you can run per node. A t3.medium can run ~17 pods, a c5.large can run ~29 pods. Plan your node sizing accordingly.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span> Traditional K8s: EKS with VPC CNI:
</span><span> +---------------------+ +---------------------+
</span><span> | VPC 10.0.0.0/16 | | VPC 10.0.0.0/16 |
</span><span> | | | |
</span><span> | EC2: 10.0.1.10 | | EC2: 10.0.1.10 |
</span><span> | RDS: 10.0.2.100 | | RDS: 10.0.2.100 |
</span><span> | | | Pod: 10.0.1.50 |
</span><span> | +---------------+ | | Pod: 10.0.1.51 |
</span><span> | | Overlay net | | | Pod: 10.0.2.20 |
</span><span> | | 192.168.0.0/16| | | (all in VPC range) |
</span><span> | | | | | |
</span><span> | | Pod: 192.168 | | | Direct routing, |
</span><span> | | Pod: 192.168 | | | no overlay needed |
</span><span> | +---------------+ | | |
</span><span> +---------------------+ +---------------------+
</span></code></pre>
<hr />
<h4 id="2-irsa-pods-get-aws-permissions-without-secrets">2. IRSA — Pods get AWS permissions without secrets</h4>
<p>Your pods need to access AWS services: read from S3, write to DynamoDB, publish to SQS. The wrong way to do this is hardcoding AWS credentials in a Secret (seriously, don't). The right way is <strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">IAM Roles for Service Accounts (IRSA)</a></strong>.</p>
<p>IRSA uses <a href="/posts/aws-2-3/#openid-connect-oidc">OpenID Connect</a> to let Kubernetes Service Accounts assume IAM roles. Here's how it works:</p>
<ol>
<li>EKS cluster has an OIDC provider endpoint</li>
<li>You create an IAM role that trusts this OIDC provider</li>
<li>You annotate a Kubernetes Service Account with the IAM role ARN</li>
<li>Pods using that Service Account automatically get temporary AWS credentials</li>
</ol>
<p><strong>Setup:</strong></p>
<p>Step 1 — Enable OIDC provider for your cluster (one-time):</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>eksctl utils associate-iam-oidc-provider \
</span><span> --cluster genomics-platform \
</span><span> --approve
</span></code></pre>
<p>Step 2 — Create an IAM policy (what AWS permissions the pod needs):</p>
<pre data-lang="json" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-json "><code class="language-json" data-lang="json"><span>{
</span><span> </span><span style="color:#d69d85;">"Version"</span><span>: </span><span style="color:#d69d85;">"2012-10-17"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Statement"</span><span>: [
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: [
</span><span> </span><span style="color:#d69d85;">"s3:GetObject"</span><span>,
</span><span> </span><span style="color:#d69d85;">"s3:PutObject"
</span><span> ],
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: </span><span style="color:#d69d85;">"arn:aws:s3:::genomics-sequencing-data/*"
</span><span> }
</span><span> ]
</span><span>}
</span></code></pre>
<p>Step 3 — Create IAM role and Service Account together:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>eksctl create iamserviceaccount \
</span><span> --name genomics-pipeline \
</span><span> --namespace genomics \
</span><span> --cluster genomics-platform \
</span><span> --attach-policy-arn arn:aws:iam::123456789012:policy/GenomicsS3ReadWritePolicy \
</span><span> --approve
</span></code></pre>
<p>Step 4 — Use the Service Account in your pod:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Pod</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">variant-analysis-job</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">serviceAccountName</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-pipeline</span><span> </span><span style="color:#608b4e;"># This pod can read/write genomics-sequencing-data
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">analyzer</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-pipeline:v1</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">env</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">AWS_REGION</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">value</span><span>: </span><span style="background-color:#282828;color:#d69d85;">eu-west-3</span><span>
</span></code></pre>
<p>Your application code just uses the standard AWS SDK — no credentials to configure:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>boto3
</span><span>
</span><span>s3 = boto3.client(</span><span style="color:#d69d85;">'s3'</span><span>) </span><span style="color:#608b4e;"># Credentials auto-discovered from IRSA
</span><span>s3.put_object(Bucket=</span><span style="color:#d69d85;">'genomics-sequencing-data'</span><span>, Key=</span><span style="color:#d69d85;">'results/patient-42.vcf'</span><span>, Body=data)
</span></code></pre>
<p><strong>Why this is brilliant</strong>: No secrets stored anywhere, credentials rotate automatically (just like EC2 instance roles from <a href="https://nskm.xyz/posts/aws-1/">Day 1</a>), and you get per-pod IAM policies — the genomics pipeline pod can access sequencing data while a separate clinical trials pod accesses a completely different bucket, all within the same cluster.</p>
<img style="display: block; margin: 0 auto; width: 600px" src="/images/irsa_eks_flow.svg" alt="" title="I am obliged to admit I'm impressed by this level of orchestration"/>
<hr />
<h4 id="3-aws-load-balancer-controller-ingress-creates-real-albs">3. AWS Load Balancer Controller — Ingress creates real ALBs</h4>
<p>When you create a Kubernetes Ingress object in EKS, you want it to provision an actual <a href="/posts/aws-1/#load-balancing">Application Load Balancer (ALB)</a>, not some pod running Nginx.</p>
<p>The <strong><a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/">AWS Load Balancer Controller</a></strong> does exactly this. Install it once in your cluster, and every time you create an Ingress, it provisions a real AWS ALB with:</p>
<ul>
<li>TLS termination using ACM certificates</li>
<li>Path-based and host-based routing</li>
<li>WAF integration for DDoS protection</li>
<li>Full CloudWatch metrics and logging</li>
</ul>
<p><strong>Installation</strong> (using Helm):</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;"># Add the EKS chart repo
</span><span>helm repo add eks https://aws.github.io/eks-charts
</span><span>helm repo update
</span><span>
</span><span style="color:#608b4e;"># Install the controller
</span><span>helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
</span><span> -n kube-system \
</span><span> --set clusterName=genomics-platform \
</span><span> --set serviceAccount.create=false \
</span><span> --set serviceAccount.name=aws-load-balancer-controller
</span></code></pre>
<p><strong>Usage</strong> — create an Ingress with ALB annotations:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">networking.k8s.io/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Ingress</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-api-ingress</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">namespace</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">annotations</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">alb.ingress.kubernetes.io/scheme</span><span>: </span><span style="background-color:#282828;color:#d69d85;">internet-facing</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">alb.ingress.kubernetes.io/target-type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">ip</span><span> </span><span style="color:#608b4e;"># Routes to pod IPs directly via VPC CNI
</span><span> </span><span style="background-color:#282828;color:#569cd6;">alb.ingress.kubernetes.io/certificate-arn</span><span>: </span><span style="background-color:#282828;color:#d69d85;">arn:aws:acm:eu-west-3:123456789012:certificate/abc123</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ingressClassName</span><span>: </span><span style="background-color:#282828;color:#d69d85;">alb</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">rules</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">host</span><span>: </span><span style="background-color:#282828;color:#d69d85;">api.genomics-platform.example.com</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">http</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">paths</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">path</span><span>: </span><span style="background-color:#282828;color:#d69d85;">/</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">pathType</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Prefix</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">backend</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">service</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">genomics-api-service</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">number</span><span>: </span><span style="color:#b5cea8;">8000
</span></code></pre>
<p>Apply it, and within 2-3 minutes, the controller creates an AWS ALB, configures target groups pointing directly to your pod IPs (thanks to VPC CNI), and updates Route 53 (if you're using <a href="https://github.com/kubernetes-sigs/external-dns">ExternalDNS</a>). Your API is now publicly accessible at <code>https://api.genomics-platform.example.com</code>, with TLS handled by ACM and traffic distributed across pods.</p>
<p><strong>This is the real and only good reason to use EKS</strong>: Kubernetes-native workflows (<code>kubectl apply -f ingress.yaml</code>) that provision real AWS infrastructure (ALBs, target groups, security groups) automatically. You get the declarative model of Kubernetes with the managed services of AWS.</p>
<img style="display: block; margin: 0 auto; width: 600px" src="/images/eks_alb_ingress_flow.svg" alt="" title="I am obliged to admit I'm impressed by this level of orchestration"/>
<hr />
<h3 id="practical-example">Practical example</h3>
<p>Let's deploy a simple web application to EKS end-to-end. We use nginx here for clarity — in <a href="https://nskm.xyz/posts/aws-6/">Day 6</a> we'll replace it with the actual genomics Django API, packaged as a Helm chart.</p>
<p><strong>Prerequisites:</strong></p>
<ul>
<li>EKS cluster running (created with <code>eksctl</code> as shown above)</li>
<li><code>kubectl</code> configured to talk to your cluster</li>
<li>AWS Load Balancer Controller installed (optional, for the Ingress part)</li>
</ul>
<p><strong>Step 1 — Create the application Deployment:</strong></p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># webapp-deployment.yaml
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">apps/v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Deployment</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">replicas</span><span>: </span><span style="color:#b5cea8;">3
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">matchLabels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">template</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">labels</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">containers</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">nginx</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">image</span><span>: </span><span style="background-color:#282828;color:#d69d85;">nginx:1.25</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">containerPort</span><span>: </span><span style="color:#b5cea8;">80
</span><span> </span><span style="background-color:#282828;color:#569cd6;">resources</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">requests</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="color:#d69d85;">"64Mi"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="color:#d69d85;">"250m"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">limits</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">memory</span><span>: </span><span style="color:#d69d85;">"128Mi"
</span><span> </span><span style="background-color:#282828;color:#569cd6;">cpu</span><span>: </span><span style="color:#d69d85;">"500m"
</span></code></pre>
<p>Apply it:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl apply -f webapp-deployment.yaml
</span><span style="color:#608b4e;"># deployment.apps/webapp created
</span><span>
</span><span>kubectl get pods
</span><span style="color:#608b4e;"># NAME READY STATUS RESTARTS AGE
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-2xqwp 1/1 Running 0 30s
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-8h4kt 1/1 Running 0 30s
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-n9p5r 1/1 Running 0 30s
</span></code></pre>
<p><strong>Step 2 — Expose the Deployment with a Service:</strong></p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># webapp-service.yaml
</span><span style="background-color:#282828;color:#569cd6;">apiVersion</span><span>: </span><span style="background-color:#282828;color:#d69d85;">v1</span><span>
</span><span style="background-color:#282828;color:#569cd6;">kind</span><span>: </span><span style="background-color:#282828;color:#d69d85;">Service</span><span>
</span><span style="background-color:#282828;color:#569cd6;">metadata</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">name</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp-service</span><span>
</span><span style="background-color:#282828;color:#569cd6;">spec</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">type</span><span>: </span><span style="background-color:#282828;color:#d69d85;">LoadBalancer</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">selector</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">app</span><span>: </span><span style="background-color:#282828;color:#d69d85;">webapp</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">ports</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">protocol</span><span>: </span><span style="background-color:#282828;color:#d69d85;">TCP</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">port</span><span>: </span><span style="color:#b5cea8;">80
</span><span> </span><span style="background-color:#282828;color:#569cd6;">targetPort</span><span>: </span><span style="color:#b5cea8;">80
</span></code></pre>
<p>Apply it:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl apply -f webapp-service.yaml
</span><span style="color:#608b4e;"># service/webapp-service created
</span><span>
</span><span>kubectl get service webapp-service
</span><span style="color:#608b4e;"># NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
</span><span style="color:#608b4e;"># webapp-service LoadBalancer 10.100.200.50 a3f2...eu-west-3.elb.amazonaws.com 80:32415/TCP 2m
</span></code></pre>
<p>EKS automatically provisions an AWS Elastic Load Balancer. After 2-3 minutes, the <code>EXTERNAL-IP</code> appears — that's your app's public endpoint.</p>
<p><strong>Step 3 — Scale the application:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl scale deployment webapp --replicas=5
</span><span style="color:#608b4e;"># deployment.apps/webapp scaled
</span><span>
</span><span>kubectl get pods
</span><span style="color:#608b4e;"># NAME READY STATUS RESTARTS AGE
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-2xqwp 1/1 Running 0 5m
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-8h4kt 1/1 Running 0 5m
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-n9p5r 1/1 Running 0 5m
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-q8w3j 1/1 Running 0 10s
</span><span style="color:#608b4e;"># webapp-7d8b9c5f6d-x7k2m 1/1 Running 0 10s
</span></code></pre>
<p>The load balancer automatically distributes traffic across all 5 pods.</p>
<p><strong>Step 4 — Update the application (rolling update):</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl set image deployment/webapp nginx=nginx:1.26
</span><span style="color:#608b4e;"># deployment.apps/webapp image updated
</span><span>
</span><span>kubectl rollout status deployment/webapp
</span><span style="color:#608b4e;"># Waiting for deployment "webapp" rollout to finish: 2 out of 5 new replicas have been updated...
</span><span style="color:#608b4e;"># deployment "webapp" successfully rolled out
</span></code></pre>
<p>K8s replaces pods one at a time, ensuring zero downtime.</p>
<p><strong>Step 5 — Check logs:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl logs -l app=webapp --tail=20
</span><span style="color:#608b4e;"># Shows logs from all webapp pods
</span><span>
</span><span>kubectl logs webapp-7d8b9c5f6d-2xqwp
</span><span style="color:#608b4e;"># Shows logs from a specific pod
</span></code></pre>
<p><strong>Step 6 — Clean up:</strong></p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>kubectl delete -f webapp-deployment.yaml
</span><span>kubectl delete -f webapp-service.yaml
</span><span style="color:#608b4e;"># Or delete everything at once:
</span><span>kubectl delete deployment,service -l app=webapp
</span></code></pre>
<p><strong>What just happened:</strong></p>
<ul>
<li>You described your desired state in YAML (3 replicas of nginx)</li>
<li>K8s scheduled pods across worker nodes</li>
<li>EKS provisioned an AWS ELB and configured it to route traffic to your pods</li>
<li>You scaled, updated, and monitored the app with simple <code>kubectl</code> commands</li>
<li>All of this ran on infrastructure defined in <a href="https://nskm.xyz/posts/aws-1/">Day 1</a> (VPC, subnets, security groups, IAM roles)</li>
</ul>
<p>This is the declarative model in action: you say <em>what</em> you want, K8s figures out <em>how</em> to make it happen.</p>
<hr />
<h3 id="eks-vs-alternatives">EKS vs. alternatives</h3>
<p>You have containerized applications and want to run them on AWS. What should you choose? Here's the honest comparison.</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Factor</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">EKS</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">ECS + Fargate</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Self-Managed K8s on EC2</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Complexity</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Medium</td>
<td style="border: 1px solid #333; padding: 10px;">Low</td>
<td style="border: 1px solid #333; padding: 10px;">High</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Setup time</strong></td>
<td style="border: 1px solid #333; padding: 10px;">15 minutes (eksctl)</td>
<td style="border: 1px solid #333; padding: 10px;">5 minutes (CloudFormation)</td>
<td style="border: 1px solid #333; padding: 10px;">Hours to days (kubeadm, Terraform)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Learning curve</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Steep (K8s expertise required)</td>
<td style="border: 1px solid #333; padding: 10px;">Gentle (AWS-native, simpler concepts)</td>
<td style="border: 1px solid #333; padding: 10px;">Very steep (K8s + infrastructure management)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Orchestration</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Full Kubernetes (industry standard)</td>
<td style="border: 1px solid #333; padding: 10px;">AWS-specific, simpler model</td>
<td style="border: 1px solid #333; padding: 10px;">Full Kubernetes, full control</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Portability</strong></td>
<td style="border: 1px solid #333; padding: 10px;">High (standard K8s, runs anywhere)</td>
<td style="border: 1px solid #333; padding: 10px;">Low (AWS-only, vendor lock-in)</td>
<td style="border: 1px solid #333; padding: 10px;">High (K8s is portable)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Infrastructure management</strong></td>
<td style="border: 1px solid #333; padding: 10px;">You manage worker nodes (or use Fargate)</td>
<td style="border: 1px solid #333; padding: 10px;">Serverless (no nodes to manage)</td>
<td style="border: 1px solid #333; padding: 10px;">You manage everything</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Ecosystem</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Massive (Helm charts, operators, CNCF projects)</td>
<td style="border: 1px solid #333; padding: 10px;">AWS services only</td>
<td style="border: 1px solid #333; padding: 10px;">Massive (K8s ecosystem)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Pricing</strong></td>
<td style="border: 1px solid #333; padding: 10px;">$73/month control plane + nodes</td>
<td style="border: 1px solid #333; padding: 10px;">Pay per task (no idle costs)</td>
<td style="border: 1px solid #333; padding: 10px;">Just EC2 costs + your time</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Scaling</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Horizontal Pod Autoscaler + Cluster Autoscaler</td>
<td style="border: 1px solid #333; padding: 10px;">Auto-scales per task</td>
<td style="border: 1px solid #333; padding: 10px;">You configure everything</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Networking</strong></td>
<td style="border: 1px solid #333; padding: 10px;">VPC CNI, complex but powerful</td>
<td style="border: 1px solid #333; padding: 10px;">AWS native, simple</td>
<td style="border: 1px solid #333; padding: 10px;">You choose (Calico, Flannel, etc.)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>IAM integration</strong></td>
<td style="border: 1px solid #333; padding: 10px;">IRSA (excellent)</td>
<td style="border: 1px solid #333; padding: 10px;">Native task roles (excellent)</td>
<td style="border: 1px solid #333; padding: 10px;">Manual (complex)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Monitoring</strong></td>
<td style="border: 1px solid #333; padding: 10px;">CloudWatch + Prometheus + third-party</td>
<td style="border: 1px solid #333; padding: 10px;">CloudWatch native</td>
<td style="border: 1px solid #333; padding: 10px;">You set up everything</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Best for</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Complex microservices, multi-cloud strategy, K8s expertise on team</td>
<td style="border: 1px solid #333; padding: 10px;">Simple containerized apps, AWS-first teams, want low ops overhead</td>
<td style="border: 1px solid #333; padding: 10px;">Full control needed, K8s expertise, cost-sensitive</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Avoid if</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Simple 2-3 container setup, no K8s experience</td>
<td style="border: 1px solid #333; padding: 10px;">Need K8s portability, complex networking requirements</td>
<td style="border: 1px solid #333; padding: 10px;">Small team, want to focus on apps not infrastructure</td>
</tr>
</tbody>
</table>
<hr />
<h2 id="my-honest-recommendation-for-a-biomedical-research-center">My honest recommendation for a biomedical research center</h2>
<p>I don't like giving recommendations. There are so many factors to consider, so many criteria to take into account. I've always believed you should start simple and, if necessary, move on to more complex options.</p>
<p>So, for most teams, <strong>start with ECS + Fargate</strong>. It's simpler, cheaper for small workloads, and gets you running quickly. As complexity grows (more services, more teams, multi-cloud needs), <em>then</em> evaluate EKS. If you already know Kubernetes or need portability: <strong>use EKS</strong>.</p>
<p>For example. If you're running batch analysis jobs (genomic processing, statistical models): <strong>ECS + Fargate</strong> is perfect. Define the job, let AWS run it, pay only for execution time. If you're building a complex platform, handling lots of requests, with web apps, APIs, databases, data pipelines, and ML inference: <strong>EKS</strong> gives you the flexibility and ecosystem to compose these pieces together.</p>
<p>Self-managed K8s only makes sense if you have specific requirements that EKS can't meet, or if you have a dedicated platform team that lives and breathes Kubernetes.</p>
<br>
<br>
<h2 id="conclusion">Conclusion</h2>
<p>Kubernetes emerged from Google's need to manage <strong><em>thousands of containers across massive infrastructure</em></strong>. It solved real problems at that <strong><em>scale</em></strong>. Today, it's becoming the <strong><em>industry standard</em></strong> because the problems it solves, orchestration, self-healing, automatic scaling, service discovery, are increasingly common. But that doesn't mean every team needs it right now. Come on man, you're not Google.</p>
<p>The real value is: if you understand Kubernetes, you understand container orchestration itself. That knowledge transfers everywhere: AWS EKS, Google GKE, Azure AKS, your own data center. The mental model is identical across platforms. That's why Kubernetes literacy is worth investing in, even if you don't deploy it immediately. Learning it now means you'll be prepared when that moment arrives.</p>
<br>
<img style="display: block; margin: 0 auto; width: 600px" src="/images/solved_problem.jpg" alt="You're welcome" title="You're welcome"/>
<br>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>This article is awfully long <em>wtf</em>! I really hope you did learn something. The topic is amazingly huge and cannot be covered in one tiny small simple article. Here are resources to continue your Kubernetes and EKS journey:</p>
<p><strong>Blog posts:</strong></p>
<ul>
<li><a href="https://www.boot.dev/blog/education/maybe-you-do-need-kubernetes/">Maybe you do need Kubernetes</a></li>
</ul>
<p><strong>Official documentation:</strong></p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/">Kubernetes documentation - concepts</a></li>
<li><a href="https://docs.aws.amazon.com/eks/latest/userguide/">Amazon EKS user guide</a></li>
<li><a href="https://www.eksworkshop.com/">Amazon EKS workshop</a></li>
<li><a href="https://aws.github.io/aws-eks-best-practices/">EKS best practices guide</a></li>
</ul>
<p><strong>Video tutorials:</strong></p>
<ul>
<li><a href="https://www.youtube.com/watch?v=H5sPGruv2yc">You don't need Kubernetes</a></li>
<li><a href="https://www.youtube.com/watch?v=s_o8dwzRlu4">Kubernetes crash course for absolute beginners</a></li>
<li><a href="https://www.youtube.com/watch?v=X48VuDVv0do">Kubernetes tutorial for beginners</a></li>
<li><a href="https://www.youtube.com/watch?v=QThadS3Soig">Getting started with Amazon EKS</a></li>
<li><a href="https://www.youtube.com/watch?v=l_iZfujb9rc">Joe Beda on co-creating Kubernetes at Google</a></li>
</ul>
<p><strong>Interactive learning:</strong></p>
<ul>
<li><a href="https://training.play-with-kubernetes.com/kubernetes-workshop/">Free browser-based K8s playground</a></li>
<li><a href="https://killercoda.com/kubernetes">KillerCoda interactive K8s tutorials</a></li>
</ul>
<p><strong>Tools to practice with:</strong></p>
<ul>
<li><a href="https://minikube.sigs.k8s.io/">Minikube</a></li>
<li><a href="https://kind.sigs.k8s.io/">Kind (Kubernetes in Docker)</a></li>
<li><a href="https://canonical.com/microk8s">MicroK8s</a></li>
<li><a href="https://k9scli.io/">k9s</a></li>
<li><a href="https://k3s.io/">k3s</a></li>
<li><a href="https://docs.k0sproject.io/stable/">k0s</a></li>
</ul>
<p><strong>Numeronyms:</strong></p>
<ul>
<li>d14n, a11y, l10n, o11y, g11n, m17n, p13n, v12n, c14n, ...</li>
</ul>
AWS (Day 2-3)2026-02-04T00:00:00+00:002026-02-04T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-2-3/ <img style="display: block; margin: 0 auto; width: 600px" src="/images/devoops.png" alt="Dev oops" title="Dev oops"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>I don't agree with this, but my employer says I don't have the right to share the source code I've written in the course of my work. <a href="https://www.youtube.com/shorts/FEYlE8pHICk">I don't ever want a problem</a>. So, I'll try to say as much as I can without divulging any specific information.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece, but if you catch any errors, please reach out—I'd be grateful for the feedback and happy to make updates!</p>
</li>
</ol>
<br>
<h2 id="hook">Hook</h2>
<p>Welcome to the second episode of this series. Already three days in the training. The pace is fast, there are many concepts to master, fatigue and exhaustion are setting in. Today, we'll talk about the security and audit systems that determine
whether your cloud infrastructure can be trusted with sensitive data. The previous episode can be found <a href="https://nskm.xyz/posts/aws-1/">here</a>.</p>
<br>
<h2 id="table-of-contents">Table of contents</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-2-3/#security-and-audit">Security and Audit</a> - How AWS enforces permissions and tracks access
<ul>
<li><a href="https://nskm.xyz/posts/aws-2-3/#iam-policies-how-aws-actually-decides">IAM Policies</a> - Policy evaluation logic and types</li>
<li><a href="https://nskm.xyz/posts/aws-2-3/#openid-connect-oidc">OpenID Connect (OIDC)</a> - Federated identity and cross-organizational access</li>
<li><a href="https://nskm.xyz/posts/aws-2-3/#kms-and-acm-encryption-and-certificates">KMS and ACM</a> - Encryption and certificate management</li>
<li><a href="https://nskm.xyz/posts/aws-2-3/#cloudtrail-who-did-what-and-when">CloudTrail</a> - Audit logging and compliance</li>
</ul>
</li>
</ol>
<br>
<br>
<h2 id="security-and-audit">Security and Audit</h2>
<p>In the <a href="https://nskm.xyz/posts/aws-1/#identity-and-access-management">previous article</a>, we introduced IAM: users, groups, roles, policies, and the principle of least privilege. That was the "who can do what" overview. Today we go deeper into the <em>how</em> — how policies are actually evaluated, how to avoid hardcoding credentials entirely (even across organizations), how to encrypt things, and how to know <em>exactly</em> what happened in your account at 3am last Tuesday.</p>
<h3 id="iam-policies-how-aws-actually-decides">IAM Policies: how AWS actually decides</h3>
<p>You write a policy, attach it to a user or role, and hope for the best. But what happens when <em>multiple</em> policies apply? When one says Allow and another says Deny? AWS follows a <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html">specific evaluation logic</a>:</p>
<ol>
<li><strong>Default deny</strong> — everything is denied unless explicitly allowed</li>
<li><strong>Explicit deny wins</strong> — if <em>any</em> policy says Deny, it's denied, period</li>
<li><strong>Allow must be present</strong> — the action must be explicitly allowed by at least one policy</li>
<li><strong>Boundaries are checked</strong> — SCPs, permission boundaries, and session policies can further restrict</li>
</ol>
<p>Think of it like this: you need a "yes" from every gatekeeper, and a single "no" from anyone overrides everything.</p>
<p><strong>Types of policies:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Policy Type</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Attached To</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Purpose</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_id-based">Identity-based</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Users, Groups, Roles</td>
<td style="border: 1px solid #333; padding: 10px;">Grant permissions to an identity</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_resource-based">Resource-based</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">S3 buckets, SQS queues, etc.</td>
<td style="border: 1px solid #333; padding: 10px;">Control who can access <em>this</em> resource</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html">Permission boundaries</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Users, Roles</td>
<td style="border: 1px solid #333; padding: 10px;">Set the <em>maximum</em> permissions an identity can have</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html">Service Control Policies</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Organization accounts</td>
<td style="border: 1px solid #333; padding: 10px;">Guardrails across entire accounts</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session">Session policies</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Temporary sessions</td>
<td style="border: 1px solid #333; padding: 10px;">Limit permissions for a specific session</td>
</tr>
</tbody>
</table>
<p><strong>Permission boundaries</strong> deserve special attention. They don't <em>grant</em> permissions — they set a ceiling on what an identity can do. Think of it as a safety net: an identity policy can allow an action, but if the permission boundary doesn't include it, the action is blocked.</p>
<p><strong>Example:</strong></p>
<ul>
<li>Permission boundary allows: <code>s3:*</code> and <code>ec2:DescribeInstances</code> (read-only)</li>
<li>Identity policy allows: <code>s3:*</code>, <code>ec2:*</code> (full EC2 access)</li>
<li><strong>Effective result</strong>: The user can do all S3 actions and only read EC2, not modify it</li>
</ul>
<blockquote>
<p>A permission boundary is a maximum limit. The effective permissions are the intersection (overlap) of:</p>
<ul>
<li>What the permission boundary allows</li>
<li>What the identity policy allows</li>
</ul>
</blockquote>
<p>This is useful when you want to let developers create their own IAM roles. You set a permission boundary that says "you can have S3 and read-only EC2," and even if a developer accidentally adds a policy that grants <code>ec2:TerminateInstances</code>, that action is blocked by the boundary. The boundary prevents privilege escalation.</p>
<p><strong>A more realistic policy example:</strong></p>
<p>The day 1 article showed a simple S3 read policy. Here's what a real-world policy looks like:</p>
<pre data-lang="json" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-json "><code class="language-json" data-lang="json"><span>{
</span><span> </span><span style="color:#d69d85;">"Version"</span><span>: </span><span style="color:#d69d85;">"2012-10-17"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Statement"</span><span>: [
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Sid"</span><span>: </span><span style="color:#d69d85;">"ReadSharedDatasets"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: [</span><span style="color:#d69d85;">"s3:GetObject"</span><span>, </span><span style="color:#d69d85;">"s3:ListBucket"</span><span>],
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: [
</span><span> </span><span style="color:#d69d85;">"arn:aws:s3:::research-datasets"</span><span>,
</span><span> </span><span style="color:#d69d85;">"arn:aws:s3:::research-datasets/shared/*"
</span><span> ],
</span><span> </span><span style="color:#d69d85;">"Condition"</span><span>: {
</span><span> </span><span style="color:#d69d85;">"StringEquals"</span><span>: {
</span><span> </span><span style="color:#d69d85;">"aws:RequestedRegion"</span><span>: </span><span style="color:#d69d85;">"eu-west-3"
</span><span> }
</span><span> }
</span><span> },
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Sid"</span><span>: </span><span style="color:#d69d85;">"UploadOwnResults"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: [</span><span style="color:#d69d85;">"s3:PutObject"</span><span>],
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: </span><span style="color:#d69d85;">"arn:aws:s3:::research-datasets/results/${aws:username}/*"
</span><span> },
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Sid"</span><span>: </span><span style="color:#d69d85;">"DenyUnencryptedUploads"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Deny"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: </span><span style="color:#d69d85;">"s3:PutObject"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: </span><span style="color:#d69d85;">"arn:aws:s3:::research-datasets/*"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Condition"</span><span>: {
</span><span> </span><span style="color:#d69d85;">"StringNotEquals"</span><span>: {
</span><span> </span><span style="color:#d69d85;">"s3:x-amz-server-side-encryption"</span><span>: </span><span style="color:#d69d85;">"aws:kms"
</span><span> }
</span><span> }
</span><span> }
</span><span> ]
</span><span>}
</span></code></pre>
<p>You read it like this (<em>Sid -> Effet -> Action -> Resource -> Condition</em>):</p>
<ul>
<li>"ReadSharedDatasets: Allow s3:GetObject, s3:ListBucket on research-datasets when in eu-west-3"</li>
<li>"UploadOwnResults: Allow s3:PutObject on own results datasets owned by current user with username"</li>
<li>"DenyUnencryptedUploads: Deny s3:PutObject on research-datasets when there is no KMS encryption"</li>
</ul>
<p>You understand it like this:</p>
<ul>
<li>"ReadSharedDatasets: Allow reading from the shared research-datasets bucket, but only from eu-west-3"</li>
<li>"UploadOwnResults: Allow uploading to your personal results folder"</li>
<li>"DenyUnencryptedUploads: Deny uploading unencrypted files to the research-datasets bucket"</li>
</ul>
<br>
Now tet's imagine the following interaction:
<p>A researcher tries to upload an unencrypted file to their results folder:</p>
<ul>
<li>Policy 2 says: "You can upload files" ✅</li>
<li>Policy 3 says: "But NOT unencrypted files" ❌</li>
<li>Result: Upload is DENIED — because Explicit deny wins (rule #2 from the evaluation logic: Explicit deny wins — if any policy says Deny, it's denied, period)</li>
</ul>
<br>
Notice three things:
<ul>
<li><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html">Conditions</a></strong> restrict <em>where</em> actions can happen (only <code>eu-west-3</code>)</li>
<li><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html">Policy variables</a></strong> like <code>${aws:username}</code> scope access per current user dynamically</li>
<li><strong>Explicit Deny blocks everything</strong> A researcher can't bypass this, even if they accidentally add a permissive policy.</li>
</ul>
<p>Why do we mention this?</p>
<p>To reinforce the critical IAM evaluation rule: <strong>"A single Deny overrides all Allows"</strong>. This is one of the most important security
principles in AWS.</p>
<br>
<p><strong>Creating an IAM Role (practical example):</strong></p>
<p>Let's say you have an EC2 instance running a Python script that needs to read from S3 and write to DynamoDB. Here's how you'd do it properly with the <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html">AWS CLI</a>:</p>
<p>Step 1 — Create the trust policy (who can assume this role):</p>
<pre data-lang="json" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-json "><code class="language-json" data-lang="json"><span>{
</span><span> </span><span style="color:#d69d85;">"Version"</span><span>: </span><span style="color:#d69d85;">"2012-10-17"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Statement"</span><span>: [
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Principal"</span><span>: { </span><span style="color:#d69d85;">"Service"</span><span>: </span><span style="color:#d69d85;">"ec2.amazonaws.com" </span><span>},
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: </span><span style="color:#d69d85;">"sts:AssumeRole"
</span><span> }
</span><span> ]
</span><span>}
</span></code></pre>
<p><strong>Reading this in plain English:</strong></p>
<blockquote>
<p>"Allow the EC2 service to assume this role (ResearchAnalysisRole)"</p>
</blockquote>
<p><strong>Breaking it down:</strong></p>
<ul>
<li><strong>Principal</strong> (<code>ec2.amazonaws.com</code>) = WHO can do this → the EC2 service</li>
<li><strong>Action</strong> (<code>sts:AssumeRole</code>) = WHAT they can do → take on this role's permissions</li>
<li><strong>Effect</strong> (<code>Allow</code>) = permission granted</li>
</ul>
<p>Step 2 — Create the role and attach the trust policy:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>aws iam create-role \
</span><span> --role-name ResearchAnalysisRole \
</span><span> --assume-role-policy-document file://trust-policy.json
</span></code></pre>
<p><strong>Key distinction:</strong></p>
<ul>
<li><strong>Trust policy</strong> (Step 1): "Who can assume this role?" → Answer: The EC2 service</li>
<li><strong>Permissions policy</strong> (Step 3): "What can this role do?" → Answer: Read S3, write to DynamoDB</li>
</ul>
<p>Step 3 — Create and attach the permissions policy:</p>
<p>First, create the <code>permissions-policy.json</code> file with the permissions the role needs:</p>
<pre data-lang="json" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-json "><code class="language-json" data-lang="json"><span>{
</span><span> </span><span style="color:#d69d85;">"Version"</span><span>: </span><span style="color:#d69d85;">"2012-10-17"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Statement"</span><span>: [
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Sid"</span><span>: </span><span style="color:#d69d85;">"ReadFromS3"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: [</span><span style="color:#d69d85;">"s3:GetObject"</span><span>, </span><span style="color:#d69d85;">"s3:ListBucket"</span><span>],
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: [
</span><span> </span><span style="color:#d69d85;">"arn:aws:s3:::research-data"</span><span>,
</span><span> </span><span style="color:#d69d85;">"arn:aws:s3:::research-data/*"
</span><span> ]
</span><span> },
</span><span> {
</span><span> </span><span style="color:#d69d85;">"Sid"</span><span>: </span><span style="color:#d69d85;">"WriteToDynamoDB"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: [</span><span style="color:#d69d85;">"dynamodb:PutItem"</span><span>, </span><span style="color:#d69d85;">"dynamodb:UpdateItem"</span><span>],
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: </span><span style="color:#d69d85;">"arn:aws:dynamodb:eu-west-3:123456789012:table/analysis-results"
</span><span> }
</span><span> ]
</span><span>}
</span></code></pre>
<p>Now attach this permissions policy to the role:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>aws iam put-role-policy \
</span><span> --role-name ResearchAnalysisRole \
</span><span> --policy-name ResearchAnalysisPermissions \
</span><span> --policy-document file://permissions-policy.json
</span></code></pre>
<p>Step 4 — Create an instance profile and attach the role:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>aws iam create-instance-profile \
</span><span> --instance-profile-name ResearchAnalysisProfile
</span><span>
</span><span>aws iam add-role-to-instance-profile \
</span><span> --instance-profile-name ResearchAnalysisProfile \
</span><span> --role-name ResearchAnalysisRole
</span></code></pre>
<p>Now any EC2 instance launched with this profile gets temporary credentials based on the ResearchAnalysisRole's permissions. automatically. No access keys to manage, no secrets to rotate, no <code>.env</code> file to accidentally commit inside the Git repository. Your Python script just calls <code>boto3.client('s3')</code> and it works, the SDK picks up the credentials from the instance metadata.</p>
<p><strong>The complete picture:</strong></p>
<pre data-lang="text" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-text "><code class="language-text" data-lang="text"><span> +------------------------------------------------------------+
</span><span> | IAM Role: ResearchAnalysisRole |
</span><span> +------------------------------------------------------------+
</span><span> | Trust Policy: |
</span><span> | "Allow EC2 service to assume this role" |
</span><span> | |
</span><span> | Permission Policy: |
</span><span> | - Allow S3: GetObject, ListBucket |
</span><span> | - Allow DynamoDB: PutItem, UpdateItem |
</span><span> +------------------------------------------------------------+
</span><span> |
</span><span> | (attached to)
</span><span> |
</span><span> +------------------------------------------------------------+
</span><span> | Instance Profile: ResearchAnalysisProfile |
</span><span> | (container that holds the role) |
</span><span> +------------------------------------------------------------+
</span><span> |
</span><span> | (attached to)
</span><span> |
</span><span> +------------------------------------------------------------+
</span><span> | EC2 Instance |
</span><span> +------------------------------------------------------------+
</span><span> | |
</span><span> | +--------------------------------------------------+ |
</span><span> | | Python Script (running on instance) | |
</span><span> | | | |
</span><span> | | import boto3 | |
</span><span> | | s3 = boto3.client('s3') | |
</span><span> | | db = boto3.client('dynamodb') | |
</span><span> | +--------------------------------------+-----------+ |
</span><span> | | |
</span><span> | +--------------------------------------+---+ |
</span><span> | | Instance Metadata Service | | |
</span><span> | | (provides temporary creds) ----------+ | |
</span><span> | | | | |
</span><span> | +--------------------------------------+---+ |
</span><span> | | |
</span><span> +-----------------------------------------+------------------+
</span><span> |
</span><span> |
</span><span> +------------+-------------------+
</span><span> | |
</span><span> +---+---+ +----+--------+
</span><span> | S3 | | DynamoDB |
</span><span> |(read) | | (write) |
</span><span> +-------+ +-------------+
</span></code></pre>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/assume-role.webp" alt="Assume role" title="Assume role"/>
<p><br><br></p>
<hr />
<h3 id="openid-connect-oidc">OpenID Connect (OIDC)</h3>
<p>So far, IAM handles identities <em>inside</em> AWS. But what about the outside world? What if your GitHub Actions pipeline needs to deploy to AWS? What if your researchers log in through your organization's Active Directory?</p>
<p><strong><a href="https://openid.net/developers/how-connect-works/">OpenID Connect (OIDC)</a></strong> is an identity layer built on top of <a href="https://oauth.net/2/">OAuth 2.0</a>. It allows external identity providers to authenticate users and services, which can then assume AWS roles. No long-lived credentials stored anywhere.</p>
<p><strong>How it works in practice:</strong></p>
<pre data-lang="text" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-text "><code class="language-text" data-lang="text"><span> Step 1: Authenticate with Identity Provider
</span><span> +---------------------------+
</span><span> | GitHub / Google / Azure AD|
</span><span> | (Identity Provider) |
</span><span> +---------------------------+
</span><span> ^
</span><span> |
</span><span> | "Login with GitHub"
</span><span> |
</span><span> +--+--+
</span><span> |User |
</span><span> +--+--+
</span><span> |
</span><span> | Step 2: Receive JWT Token
</span><span> | (cryptographic proof of identity)
</span><span> v
</span><span> +---------------------------+
</span><span> | JWT Token |
</span><span> | Contains: |
</span><span> | - Repository name |
</span><span> | - Branch |
</span><span> | - Commit SHA |
</span><span> | - Timestamp |
</span><span> +---------------------------+
</span><span> |
</span><span> | Step 3: Send token to AWS STS
</span><span> v
</span><span> +---------------------------+
</span><span> | AWS STS |
</span><span> | (Security Token Service) |
</span><span> +---------------------------+
</span><span> |
</span><span> | Step 4: STS checks trust policy
</span><span> | "Is this token from the repo
</span><span> | I trust? YES -> OK"
</span><span> |
</span><span> v
</span><span> +---------------------------+
</span><span> | Temporary AWS Credentials |
</span><span> | - Access Key ID |
</span><span> | - Secret Access Key |
</span><span> | - Session Token |
</span><span> | - Expires in 1 hour |
</span><span> +---------------------------+
</span></code></pre>
<p><strong>The flow:</strong></p>
<ol>
<li><strong>User/Service authenticates</strong> with an external identity provider (GitHub, Google, Azure AD, corporate LDAP)</li>
<li><strong>Provider issues a JSON Web Token</strong>, a cryptographic <a href="https://jwt.io/introduction">proof</a> containing metadata (repository, branch, user identity, etc.)</li>
<li><strong>Token is sent to AWS STS (Security Token Service)</strong>, <a href="https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html">STS</a> examines it</li>
<li><strong>STS checks the trust policy</strong> — "Is this token from a trusted source?" If yes, temporary credentials are issued</li>
<li><strong>Credentials expire automatically</strong> — typically 1 hour, so no manual cleanup needed</li>
</ol>
<p><strong>Real-world example — GitHub Actions deploying to AWS:</strong></p>
<p>Instead of storing AWS access keys as GitHub secrets, you configure an <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html">OIDC identity provider in AWS</a>:</p>
<pre data-lang="yaml" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#608b4e;"># .github/workflows/deploy.yml
</span><span style="background-color:#282828;color:#569cd6;">jobs</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">deploy</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">permissions</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">id-token</span><span>: </span><span style="background-color:#282828;color:#d69d85;">write</span><span> </span><span style="color:#608b4e;"># Required for OIDC
</span><span> </span><span style="background-color:#282828;color:#569cd6;">contents</span><span>: </span><span style="background-color:#282828;color:#d69d85;">read</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">steps</span><span>:
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">uses</span><span>: </span><span style="background-color:#282828;color:#d69d85;">aws-actions/configure-aws-credentials@v4</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">with</span><span>:
</span><span> </span><span style="background-color:#282828;color:#569cd6;">role-to-assume</span><span>: </span><span style="background-color:#282828;color:#d69d85;">arn:aws:iam::123456789012:role/GitHubDeployRole</span><span>
</span><span> </span><span style="background-color:#282828;color:#569cd6;">aws-region</span><span>: </span><span style="background-color:#282828;color:#d69d85;">eu-west-3</span><span>
</span><span> - </span><span style="background-color:#282828;color:#569cd6;">run</span><span>: </span><span style="background-color:#282828;color:#d69d85;">aws s3 sync ./build s3://my-app-bucket</span><span>
</span></code></pre>
<p><strong>Security: What if someone forks your repo?</strong></p>
<p>If someone forks your repo, they can't assume your role. Here's why:</p>
<ol>
<li><strong>The Github OIDC token includes metadata</strong>: repository name, branch, commit SHA, etc.</li>
<li><strong>Your trust policy is specific</strong>: it explicitly checks: "Only allow tokens from <code>organization/original-repo-name</code>"</li>
<li><strong>The fork has different metadata</strong>: when the fork's workflow runs, the token says <code>fork-owner/forked-repo</code></li>
<li><strong>The policy rejects it</strong>: AWS compares: <code>fork-owner/forked-repo</code> ≠ <code>organization/original-repo-name</code> → Denied</li>
<li><strong>Result: No credentials, no access</strong> — Even if the attacker has your exact workflow file, the OIDC token won't match the trust policy</li>
</ol>
<p><strong>Why this matters for a biomedical research center:</strong></p>
<p>Researchers often come from partner institutions, universities, or international organizations. Instead of creating and managing IAM users for each collaborator, you can federate identity through their home institution's identity provider. They log in with their university credentials, get temporary AWS access scoped to exactly what they need, and when their collaboration ends, you revoke nothing — their access simply stops when the trust relationship is removed.</p>
<img style="display: block; margin: 0 auto; width: 210px" src="/images/security-check.webp" alt="Security check" title="Security check"/>
<p><br><br></p>
<hr />
<h3 id="kms-and-acm-encryption-and-certificates">KMS and ACM: encryption and certificates</h3>
<p><strong><a href="https://docs.aws.amazon.com/kms/latest/developerguide/overview.html">AWS KMS (Key Management Service)</a></strong></p>
<p>Manages encryption keys for your data. If you're storing patient data, genomic sequences, or any sensitive research data, encryption isn't optional — it's a compliance requirement (HIPAA, GDPR, etc.).</p>
<p><strong>Key concepts:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Concept</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">What it does</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#kms_keys">Customer Master Key (CMK)</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">The top-level key. Never leaves AWS. You control who can use it. IAM policies define access.</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys">Data key</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">Generated by KMS, used to actually encrypt your data. You use it locally, then discard it.</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html">Key policies</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">IAM-like policies controlling who can use or manage the key. Auditable via CloudTrail.</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong><a href="https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html">Automatic rotation</a></strong></td>
<td style="border: 1px solid #333; padding: 10px;">AWS rotates key material annually. Old data remains decryptable. Compliance requirement.</td>
</tr>
</tbody>
</table>
<p><strong>How envelope encryption works:</strong></p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>Step 1: Request data key from KMS
</span><span>Application --> KMS: "Give me a data key"
</span><span>
</span><span>Step 2: KMS returns both plaintext and encrypted key
</span><span>KMS --> Application:
</span><span> - Plaintext key (256-bit AES key)
</span><span> - Encrypted key (encrypted with CMK)
</span><span>
</span><span>Step 3: Encrypt data locally with plaintext key
</span><span>Application:
</span><span> - Encrypts patient data with plaintext key
</span><span> - Discards plaintext key from memory
</span><span>
</span><span>Step 4: Store encrypted data + encrypted key
</span><span>Database stores:
</span><span> - Encrypted patient data (gigabytes)
</span><span> - Encrypted data key (tiny, ~200 bytes)
</span><span>
</span><span>Step 5: To decrypt later
</span><span>Application: "KMS, decrypt this key for me"
</span><span>KMS: Decrypts it, returns plaintext key
</span><span>Application: Uses plaintext key to decrypt patient data
</span></code></pre>
<p><strong>Why this design?</strong> Sending gigabytes of sensitive data to KMS would be slow, expensive, and create a bottleneck. Instead, KMS only handles small cryptographic keys, and the heavy encryption happens locally in your application. Best of both worlds: security + performance.</p>
<p><strong>Practical example — patient records:</strong></p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>1. Patient uploads medical scan (500 MB)
</span><span>2. You request a data key from KMS (costs ~$0.03)
</span><span>3. Application encrypts scan with plaintext key
</span><span>4. Plaintext key is immediately discarded
</span><span>5. Database stores:
</span><span> - Encrypted scan (500 MB, unreadable)
</span><span> - Encrypted data key (200 bytes, small)
</span><span>6. The CMK stays locked in KMS (never exported, never leaves AWS)
</span><span>7. To decrypt: send encrypted key to KMS, get plaintext key
</span><span>8. Audit trail in CloudTrail: "Who decrypted what and when"
</span></code></pre>
<p><strong>AWS-managed vs. Customer-managed keys:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Type</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Cost</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Control</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Use when</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>AWS-managed</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Free</td>
<td style="border: 1px solid #333; padding: 10px;">Limited (AWS rotates annually)</td>
<td style="border: 1px solid #333; padding: 10px;">Most AWS services (S3, RDS, DynamoDB by default)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Customer-managed</strong></td>
<td style="border: 1px solid #333; padding: 10px;">~$1/month + API calls</td>
<td style="border: 1px solid #333; padding: 10px;">Full (you choose rotation, who can access, audit trail)</td>
<td style="border: 1px solid #333; padding: 10px;">Compliance-heavy (HIPAA, GDPR, biomedical data)</td>
</tr>
</tbody>
</table>
<p>For biomedical research with patient data, <strong>customer-managed keys are typically required</strong> because auditors need to see exactly who accessed the encryption keys and when (CloudTrail).</p>
<hr />
<p><strong><a href="https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html">AWS ACM (Certificate Manager)</a></strong></p>
<p>Handles TLS/SSL certificates for your domains. If you've wrestled with <a href="https://letsencrypt.org/">Let's Encrypt</a> renewals, ACM is the managed equivalent.</p>
<p><strong>Public certificates (Free):</strong></p>
<ul>
<li>Domain validation: ACM verifies you own the domain</li>
<li>Use with: ALB, CloudFront, API Gateway</li>
<li>Automatic renewal: No more "certificate expired at 3am" alerts</li>
<li>Wildcard support: <code>*.example.com</code> covers all subdomains</li>
</ul>
<p><strong>Private CA (Paid):</strong></p>
<ul>
<li>For internal services that need TLS but aren't exposed to the internet</li>
<li>Example: Your research database server communicates with Python scripts using mutual TLS</li>
<li>Cost: ~$400/month for the CA, plus per-certificate fees</li>
<li>Use when: Internal services need encrypted communication, but you control both client and server</li>
</ul>
<p><strong>The tradeoff:</strong></p>
<ul>
<li><strong>ACM public certificates:</strong> Free, automatic, but only work with AWS services</li>
<li><strong>Let's Encrypt:</strong> Free, but you manage renewal automation yourself</li>
<li><strong>ACM private CA:</strong> Paid, but manages all internal TLS automatically</li>
</ul>
<p>For biomedical research centers: use <strong>free public certificates for your website</strong> (ACM + ALB), and use <strong>HTTP/2 between internal services</strong> (encryption handled by VPC itself, no need for certificates).</p>
<br>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/crypto.png" alt="Speak louder for those at the back of the room" title="Speak louder for those at the back of the room"/>
<p><br><br></p>
<hr />
<h3 id="cloudtrail-who-did-what-and-when">CloudTrail: who did what, and when</h3>
<p><strong><a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html">AWS CloudTrail</a></strong> records every API call made in your AWS account. Every. Single. One. Someone launched an EC2 instance? Logged. Someone changed a security group? Logged. Someone tried to access an S3 bucket and was denied? Logged.</p>
<p><strong>What CloudTrail captures:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Field</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Example</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Who</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>arn:aws:iam::123456789012:user/jdoe</code></td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>What</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>RunInstances</code>, <code>PutBucketPolicy</code>, <code>DeleteObject</code></td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>When</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>2026-02-04T14:32:17Z</code></td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Where</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>eu-west-3</code>, source IP <code>203.0.113.42</code></td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>How</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Console, CLI, SDK, or assumed role</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Result</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Success or failure (with error code)</td>
</tr>
</tbody>
</table>
<p><strong>Event types:</strong></p>
<ul>
<li><strong>Management events</strong>: Control plane operations (creating resources, changing configurations). Logged by default.</li>
<li><strong>Data events</strong>: Data plane operations (reading S3 objects, invoking Lambda functions). <em>Not</em> logged by default — you must explicitly enable them. They're high-volume and cost money.</li>
<li><strong>Insights events</strong>: CloudTrail analyzes your patterns and flags unusual activity (sudden spike in API calls, unusual error rates).</li>
</ul>
<p><strong>Practical example — investigating a security incident:</strong></p>
<p>A researcher reports they can't access their S3 bucket anymore. You suspect someone changed the bucket policy. With CloudTrail and <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html">CloudWatch Logs Insights</a>:</p>
<pre data-lang="sql" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-sql "><code class="language-sql" data-lang="sql"><span>fields @</span><span style="color:#569cd6;">timestamp</span><span>, </span><span style="color:#b4cea8;">userIdentity</span><span>.</span><span style="color:#b4cea8;">arn</span><span>, eventName, </span><span style="color:#b4cea8;">requestParameters</span><span>.</span><span style="color:#b4cea8;">bucketName
</span><span>| filter eventSource = </span><span style="color:#d69d85;">"s3.amazonaws.com"
</span><span>| filter eventName </span><span style="color:#569cd6;">like /</span><span>PutBucket</span><span style="color:#569cd6;">/
</span><span>| filter </span><span style="color:#b4cea8;">requestParameters</span><span>.</span><span style="color:#b4cea8;">bucketName </span><span>= </span><span style="color:#d69d85;">"research-datasets"
</span><span>| sort @</span><span style="color:#569cd6;">timestamp desc
</span><span>| </span><span style="color:#569cd6;">limit </span><span style="color:#b5cea8;">20
</span></code></pre>
<p>This query shows you exactly who modified the bucket policy, when, and from where. No more guessing, no more "it wasn't me." The audit trail is immutable.</p>
<p><strong>Best practices:</strong></p>
<ol>
<li><strong>Enable CloudTrail in all regions</strong> — attacks often target regions you're not watching</li>
<li><strong>Send logs to a centralized S3 bucket</strong> in a dedicated security account — so attackers can't delete their tracks even if they compromise a workload account</li>
<li><strong>Enable log file integrity validation</strong> — CloudTrail can detect if someone tampered with log files</li>
<li><strong>Set up <a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html">CloudWatch alarms</a></strong> for critical events: root account login, IAM policy changes, security group modifications, console sign-in failures</li>
</ol>
<p><strong>The traditional infrastructure equivalent:</strong></p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">CloudTrail Feature</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Traditional Equivalent</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Management events</td>
<td style="border: 1px solid #333; padding: 10px;"><code>auditd</code> on Linux, syslog for config changes</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Data events</td>
<td style="border: 1px solid #333; padding: 10px;">Application-level access logs (Apache, Nginx logs)</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Insights</td>
<td style="border: 1px solid #333; padding: 10px;">Custom anomaly detection scripts, SIEM correlation rules</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Log integrity</td>
<td style="border: 1px solid #333; padding: 10px;"><code>AIDE</code>, <code>Tripwire</code>, signed log shipping</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;">Multi-region logging</td>
<td style="border: 1px solid #333; padding: 10px;">Centralized syslog server, rsyslog forwarding</td>
</tr>
</tbody>
</table>
<p><strong>In the biomedical research context:</strong></p>
<p>CloudTrail is <em>required</em> for compliance. HIPAA mandates audit trails for access to protected health information. GDPR requires you to demonstrate who accessed personal data and why. CloudTrail, combined with <a href="https://docs.aws.amazon.com/config/latest/developerguide/WhatIsConfig.html">AWS Config</a> (which tracks <em>resource</em> configuration changes over time), gives you a complete audit story.</p>
<hr />
<br>
<br>
<h2 id="equivalence-in-the-linux-world">Equivalence in the Linux world</h2>
<p>The concepts we've covered aren't AWS innovations. They're <strong>fundamental security patterns</strong> that exist in every operating system. Here's how AWS services map to traditional Linux/Unix equivalents:</p>
<table style="border-collapse: collapse; border: 1px solid #333; width: 100%;">
<thead>
<tr style="border: 1px solid #333;">
<th style="border: 1px solid #333; padding: 10px; text-align: left;">AWS Service</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Linux/Unix Equivalent</th>
<th style="border: 1px solid #333; padding: 10px; text-align: left;">Purpose</th>
</tr>
</thead>
<tbody>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>IAM Policies</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>/etc/sudoers</code>, file permissions (rwx), RBAC</td>
<td style="border: 1px solid #333; padding: 10px;">Define who can do what</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Permission Boundaries</strong></td>
<td style="border: 1px solid #333; padding: 10px;">AppArmor, SELinux policies</td>
<td style="border: 1px solid #333; padding: 10px;">Set maximum permissions ceiling</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>OIDC (Federated Identity)</strong></td>
<td style="border: 1px solid #333; padding: 10px;">LDAP, Kerberos, PAM modules</td>
<td style="border: 1px solid #333; padding: 10px;">External identity provider authentication</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>Trust Policy</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>/etc/sudoers</code>, PAM rules</td>
<td style="border: 1px solid #333; padding: 10px;">Define who can assume a role/user</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>KMS (Encryption Keys)</strong></td>
<td style="border: 1px solid #333; padding: 10px;">GPG, OpenSSL, PKCS#11, HSM</td>
<td style="border: 1px solid #333; padding: 10px;">Manage encryption keys centrally</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>ACM (Certificates)</strong></td>
<td style="border: 1px solid #333; padding: 10px;">Let's Encrypt + certbot, OpenSSL</td>
<td style="border: 1px solid #333; padding: 10px;">Manage TLS/SSL certificates</td>
</tr>
<tr style="border: 1px solid #333;">
<td style="border: 1px solid #333; padding: 10px;"><strong>CloudTrail (Audit)</strong></td>
<td style="border: 1px solid #333; padding: 10px;"><code>auditd</code>, <code>syslog</code>, <code>journalctl</code></td>
<td style="border: 1px solid #333; padding: 10px;">Immutable audit logs of system activity</td>
</tr>
</tbody>
</table>
<p><strong>Why AWS then?</strong> These equivalents work great on Linux, but AWS eliminates the plumbing work, automates, standardizes, and integrates these patterns across hundreds of services. AWS provides enterprise-grade reliability.</p>
<br>
<br>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>If Day 1 was about "where and what," Day 2-3 has been about "how and who." Security and audit should not be something you think about <em>after</em>. They're the foundation of any production infrastructure, especially even more for people handling sensitive data like what we do with <a href="/posts/syndromic-surveillance/">4S</a>.</p>
<p>The patterns here generalize beyond AWS: whether you're running on-premises or on another cloud, the principles remain the same:</p>
<ul>
<li><strong>Default deny, explicit allow</strong>: the principle of least privilege applied everywhere</li>
<li><strong>Separate the identity layer</strong>: authentication (who you are) from authorization (what you can do)</li>
<li>**Keep an immutable audit trail: you can't defend what you can't prove</li>
<li><strong>Encrypt by default</strong>: both in transit and at rest</li>
<li><strong>Use temporary credentials</strong>: no long-lived secrets scattered across your infrastructure</li>
</ul>
<p>I don't think I have fully understood and mastered everything. I don't think I can explore everything in a single article. If you want to dive deeper into AWS security and audit, I strongly recommend the following links:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html">AWS Identity and Access Management Best Practices</a></li>
<li><a href="https://aws.amazon.com/architecture/security-identity-compliance/">AWS Security Best Practices</a></li>
<li><a href="https://aws.amazon.com/compliance/hipaa-compliance/">HIPAA Compliance on AWS</a></li>
<li><a href="https://aws.amazon.com/compliance/gdpr-center/">GDPR Compliance on AWS</a></li>
<li><a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html">CloudTrail User Guide</a></li>
<li><a href="https://docs.aws.amazon.com/kms/latest/developerguide/best-practices.html">KMS Best Practices</a></li>
<li><a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/">AWS Security Reference Architecture (SRA)</a></li>
<li><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html">OpenID Connect on AWS</a></li>
<li><a href="https://owasp.org/www-project-top-ten/">The OWASP Top 10 and how cloud addresses it</a></li>
<li><a href="https://aws.amazon.com/security/zero-trust/">Zero Trust Architecture</a></li>
</ul>
AWS (Day 1)2026-02-02T00:00:00+00:002026-02-02T00:00:00+00:00
Unknown
https://nskm.xyz/posts/aws-1/ <img style="display: block; margin: 0 auto; width: 400px" src="/images/cloud-infra.jpg" alt="infrastructure" title="infrastructure"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>I don't agree with this, but my employer says I don't have the right to share the source code I've written in the course of my work. <a href="https://www.youtube.com/shorts/FEYlE8pHICk">I don't ever want a problem</a>. So, I'll try to say as much as I can without divulging any specific information.</p>
</li>
<li>
<p>If you are a colleague and recognize the situation described below, remember: I'm critiquing the system, not the people.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece, but if you catch any errors, please reach out—I'd be grateful for the feedback and happy to make updates!</p>
</li>
<li>
<p>Render unto Caesar what is Caesar's, Claude helped me find up-to-date documentation links and he also helped beautify all the diagrams used (and not used) in this article.</p>
</li>
</ol>
<br>
<br>
<h2 id="hook">Hook</h2>
<p>One of the joys of corporate life: you're politely asked what training would help you be more productive, and a few months later, you're enrolled in something completely unrelated to your request.</p>
<p>I've been asked to take a course called "AWS Automation and Observability." Of course, I only have one choice: accept. Even though I wasn't sure it would be immediately relevant to my current work.</p>
<p>Training you didn't choose <em>(especially when you already have some technical background)</em> can feel misaligned with your current interests. When the content covers tools like <a href="https://i.giphy.com/pPhyAv5t9V8djyRFJH.webp">Jenkins</a> without discussing more modern CI/CD approaches, the relevance can feel limited.</p>
<p>There's a certain irony in being told:</p>
<blockquote>
<p>« Il est strictement interdit d'enregistrer, de filmer ou de fixer par quelque moyen que ce soit les sessions de formation. Les supports doivent rester confidentiels. »</p>
</blockquote>
<p>...when the material itself is widely available in public documentation, and when <em>(at least for day 0)</em>, there was no instructor-specific examples or custom material. I respect the instructor's knowledge of the subject, <em>he clearly knows his stuff</em>, but I felt the format, delivery, and relevance could have been better tailored to the audience.</p>
<p>Perhaps the problem isn't the instructor but the constraints he was working under: limited time, the fact that there were too many people with different levels in the same class, or budget restrictions. In an ideal world, training would be more engaging, current, and tailored to diverse experience levels. There were opportunities to improve on these dimensions.</p>
<p>Since there's no escaping it, might as well make the most of it and stay in a <a href="https://www.youtube.com/watch?v=m7XZyj6Ru9w">good mood</a>. Let's talk about what I wish the first day had covered and <em>how</em> it should have been covered.</p>
<h2 id="aws">AWS</h2>
<h3 id="intro">Intro</h3>
<p><strong><a href="https://aws.amazon.com/">Amazon Web Services (AWS)</a></strong> is Amazon's cloud computing platform, launched in 2006 with two initial services: <a href="https://aws.amazon.com/s3/">S3 (Simple Storage Service)</a> and <a href="https://aws.amazon.com/ec2/">EC2 (Elastic Compute Cloud)</a>. The idea emerged from Amazon's internal infrastructure challenges—they had built robust, scalable systems to handle their e-commerce peaks, and realized they could rent that infrastructure to others.</p>
<p>Today, AWS offers <a href="https://aws.amazon.com/products/">200+ services</a> spanning compute, storage, databases, machine learning, networking, and more. It holds roughly 31% of the global cloud market share (as of 2024), making it the largest cloud provider. <em>Phew</em>.</p>
<p>In Africa, the <a href="https://aws.amazon.com/local/africa/cape-town/">Cape Town region (af-south-1)</a> was launched in 2020. It is AWS's first and currently only African region. There's no dedicated region yet for West Africa specifically. Users typically connect to eu-west-3 (Paris) or af-south-1 (Cape Town) depending on latency requirements.</p>
<p>AWS does also have :</p>
<ul>
<li><a href="https://aws.amazon.com/about-aws/whats-new/2023/06/aws-edge-location-nigeria/">Edge locations (CloudFront POPs) in Lagos and Nairobi</a> for content delivery.</li>
<li><a href="https://aws.amazon.com/about-aws/whats-new/2025/04/aws-wavelength-zone-dakar/">Wavelength zone in Dakar Senegal</a>, a new type of infrastructure designed to run workloads that require low latency or edge resiliency.</li>
</ul>
<h3 id="alternatives-the-hyperscalers">Alternatives: the hyperscalers</h3>
<p><strong>Hyperscalers</strong> are cloud providers operating at <em>massive</em> scale; we are talking about data centers across multiple continents, serving <em>millionZ</em> of customers simultaneously. The <em>"Big 3"</em> dominate the market:</p>
<table><thead><tr><th>Provider </th><th>Share</th><th>Strengths</th><th>Best For</th></tr></thead><tbody>
<tr><td><strong><a href="https://aws.amazon.com/">AWS</a></strong></td><td>~31%</td><td>Widest service catalog, mature ecosystem</td><td>General purpose, enterprise, startups</td></tr>
<tr><td><strong><a href="https://azure.microsoft.com/">Azure</a></strong></td><td>~24%</td><td>Windows integration, hybrid cloud</td><td>Microsoft shops, hybrid deployments</td></tr>
<tr><td><strong><a href="https://cloud.google.com/">GCP</a></strong></td><td>~11%</td><td>Data analytics, ML/AI (TensorFlow), Kubernetes </td><td>Data-heavy workloads, ML projects</td></tr>
</tbody></table>
<p><strong>Notable alternatives:</strong></p>
<ul>
<li><strong><a href="https://www.digitalocean.com/">DigitalOcean</a> / <a href="https://www.linode.com/">Linode</a> / <a href="https://www.vultr.com/">Vultr</a></strong> — Simpler, developer-friendly, cheaper for small workloads</li>
<li><strong><a href="https://www.ovhcloud.com/">OVHcloud</a></strong> — European provider, good for GDPR compliance, competitive pricing</li>
<li><strong><a href="https://www.scaleway.com/">Scaleway</a></strong> — French provider, strong EU presence, interesting ARM offerings</li>
<li><strong><a href="https://www.hetzner.com/">Hetzner</a></strong> — German provider, excellent price-to-performance ratio, popular for self-hosted projects</li>
</ul>
<p>The hyperscaler choice often comes down to existing tooling, team expertise, and specific service needs rather than raw capability. They all solve similar problems.</p>
<p>As an advocate for the <a href="https://www.gnu.org/philosophy/free-software-intro.en.html">free software movement</a>, what is my advice ?</p>
<p>GCP: While proprietary, Google has historically <a href="https://opensource.google/projects">opened up key technologies</a> such as Kubernetes and <a href="https://github.com/google">many others</a> to promote interoperability and reduce vendor lock-in.</p>
<p><a href="https://www.redhat.com/en/technologies/cloud-computing/openshift">Red Hat OpenShift (on any cloud)</a>: Not a <a href="https://www.redhat.com/en/topics/cloud-computing/what-is-a-hyperscaler">hyperscaler</a> per se, but using this platform on a cloud infrastructure allows you to maintain a free and portable software layer between different providers.</p>
<h3 id="why-would-a-biomedical-research-center-use-aws">Why would a biomedical research center use AWS ?</h3>
<p>For an organization handling sensitive health data and research workloads:</p>
<ul>
<li><strong>Compliance</strong>: HIPAA, GDPR-ready configurations</li>
<li><strong>Burst capacity</strong>: spin up hundreds of CPUs for genomic analysis, pay only for what you use</li>
<li><strong>Global collaboration</strong>: share datasets across continents with proper access controls</li>
<li><strong>Focus on research</strong>: less time managing servers, more time on science</li>
</ul>
<p><br><br></p>
<h2 id="table-of-contents">Table of contents</h2>
<ol>
<li><a href="https://nskm.xyz/posts/aws-1/#fundamentals">Fundamentals</a></li>
<li><a href="https://nskm.xyz/posts/aws-1/#identity-and-access-management">Identity and Access Management</a></li>
<li><a href="https://nskm.xyz/posts/aws-1/#network">Network</a>
<ul>
<li>VPC basics (CIDR, Subnets)</li>
<li>Gateways (IGW, NAT)</li>
<li>Hybrid connectivity</li>
<li>Network Security</li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/aws-1/#load-balancing">Load Balancing</a></li>
<li><a href="https://nskm.xyz/posts/aws-1/#compute-services">Compute Services</a></li>
<li><a href="https://nskm.xyz/posts/aws-1/#storage">Storage</a></li>
<li><a href="https://nskm.xyz/posts/aws-1/#observability">Observability</a></li>
</ol>
<p><br><br></p>
<h2 id="aws-concepts-traditional-infrastructure">AWS concepts → Traditional infrastructure</h2>
<p>If you've been managing Linux servers, you already know most of these concepts. AWS just wraps them in managed services with different names:</p>
<table><thead><tr><th>AWS Service/Concept</th><th>Traditional Equivalent</th><th>What's Different?</th></tr></thead><tbody>
<tr><td><strong>EC2</strong></td><td>Physical server, VM (KVM/VMware)</td><td>Rent by the hour, no hardware to buy</td></tr>
<tr><td><strong>VPC</strong></td><td>Network subnet, VLAN</td><td>Software-defined, fully isolated per customer</td></tr>
<tr><td><strong>Security Groups</strong></td><td><code>iptables</code> rules (instance-level)</td><td>Stateful, attached to network interfaces</td></tr>
<tr><td><strong>Network ACLs</strong></td><td><code>iptables</code> at router/subnet level</td><td>Stateless, numbered rules, processed in order</td></tr>
<tr><td><strong>S3</strong></td><td>File server (NFS, Samba), MinIO</td><td>Object storage (not filesystem), infinite scale, HTTP API</td></tr>
<tr><td><strong>EBS</strong></td><td>Hard drive, LVM volume</td><td>Network-attached block storage, point-in-time snapshots</td></tr>
<tr><td><strong>EFS</strong></td><td>NFS share</td><td>Managed NFS, auto-scaling, multi-AZ</td></tr>
<tr><td><strong>RDS</strong></td><td>PostgreSQL server you installed</td><td>AWS manages backups, patches, replication, HA</td></tr>
<tr><td><strong>Load Balancer (ALB/NLB)</strong></td><td>Nginx, HAProxy, Apache <code>mod_proxy</code></td><td>Fully managed, auto-scales, integrated health checks</td></tr>
<tr><td><strong>Auto Scaling</strong></td><td>Custom bash scripts watching <code>top</code></td><td>Automatic instance scaling based on CloudWatch metrics</td></tr>
<tr><td><strong>IAM</strong></td><td><code>/etc/passwd</code>, <code>sudo</code>, LDAP/Active Directory</td><td>Controls AWS API access, not just OS users</td></tr>
<tr><td><strong>IAM Roles</strong></td><td>Service accounts, <code>systemd</code> user credentials</td><td>Temporary credentials that auto-rotate</td></tr>
<tr><td><strong>CloudWatch Metrics</strong></td><td>Prometheus, Nagios, Zabbix, Grafana</td><td>Pre-integrated with all AWS services</td></tr>
<tr><td><strong>CloudWatch Logs</strong></td><td><code>syslog</code>, <code>journald</code>, ELK stack</td><td>Centralized log aggregation, SQL-like queries</td></tr>
<tr><td><strong>CloudWatch Alarms</strong></td><td>Nagios alerts, <code>monit</code>, custom scripts</td><td>Trigger actions (scaling, notifications, Lambda)</td></tr>
<tr><td><strong>CloudTrail</strong></td><td><code>auditd</code>, <code>/var/log/audit/audit.log</code></td><td>Every AWS API call logged (who did what, when)</td></tr>
<tr><td><strong>Lambda</strong></td><td>Cron jobs + shell scripts</td><td>Event-driven functions, no server to manage</td></tr>
<tr><td><strong>Route 53</strong></td><td>BIND, <code>dnsmasq</code>, PowerDNS</td><td>Managed DNS with health checks and routing policies</td></tr>
<tr><td><strong>Regions</strong></td><td>Multiple data center locations</td><td>Completely isolated, compliance/data residency boundaries</td></tr>
<tr><td><strong>Availability Zones</strong></td><td>Separate racks/power/networking in same DC</td><td>Physical separation within region, low-latency links</td></tr>
</tbody></table>
<p><strong>The trade-off</strong>: You give up control (can't SSH into the Load Balancer) in exchange for less maintenance (AWS patches it for you). Whether that's worth it depends on your team size, expertise, and what you're trying to build. And if you understood everything inside the table above, you can stop here, I'm serious :\</p>
<p><br><br></p>
<h2 id="want-to-know-more-okay">Want to know more ? Okay</h2>
<h3 id="fundamentals">Fundamentals</h3>
<p><strong><a href="https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html">Regions</a></strong> are geographically isolated clusters of data centers. Each region (e.g., <code>eu-west-3</code> for Paris, <code>us-east-1</code> for N. Virginia) operates independently. You choose a region based on:</p>
<ul>
<li><strong>Latency</strong> — closer to your users = faster response</li>
<li><strong>Compliance</strong> — data residency laws may require specific locations</li>
<li><strong>Service availability</strong> — not all services exist in all regions</li>
<li><strong>Pricing</strong> — costs vary by region</li>
</ul>
<p><strong><a href="https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-availability-zones.html">Availability Zones (AZs)</a></strong> are physically separate data centers within a region, connected by low-latency links. Deploying across multiple Zones provides fault tolerance: if one data center fails, your application survives.</p>
<p><strong><a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html">Organizations</a></strong> is a service designed for central governance and management of multiple AWS accounts. They are useful for:</p>
<ul>
<li>Isolating environments (dev/staging/prod)</li>
<li>Separating billing by department or project</li>
<li>Applying security policies across accounts</li>
<li>Consolidated billing with volume discounts</li>
</ul>
<h3 id="identity-and-access-management">Identity and Access Management</h3>
<p>Before you start spinning up resources, you need to understand <strong>who can do what</strong> in your AWS account. This is where <strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html">IAM (Identity and Access Management)</a></strong> comes in.</p>
<p>IAM is AWS's authentication and authorization service. It controls who can access your AWS resources and what actions they can perform. Get this wrong, and you'll either lock yourself out or leave your infrastructure wide open to attackers.</p>
<p><strong>Core IAM concepts:</strong></p>
<table><thead><tr><th>Component</th><th>Purpose</th><th>When to Use</th></tr></thead><tbody>
<tr><td><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html">Users</a></strong></td><td>Individual identities with permanent credentials</td><td>Human access via Console/CLI</td></tr>
<tr><td><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html">Groups</a></strong></td><td>Collections of users with shared permissions</td><td>Organizing users by function (developers, ops, analysts)</td></tr>
<tr><td><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html">Roles</a></strong></td><td>Temporary credentials that can be assumed</td><td>Services, applications, cross-account access</td></tr>
<tr><td><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html">Policies</a></strong></td><td>JSON documents defining permissions</td><td>Attach to users, groups, or roles</td></tr>
</tbody></table>
<p><strong><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html">IAM Policies</a></strong> are JSON documents that specify allowed or denied actions. Example:</p>
<pre data-lang="json" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-json "><code class="language-json" data-lang="json"><span>{
</span><span> </span><span style="color:#d69d85;">"Version"</span><span>: </span><span style="color:#d69d85;">"2012-10-17"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Statement"</span><span>: [{
</span><span> </span><span style="color:#d69d85;">"Effect"</span><span>: </span><span style="color:#d69d85;">"Allow"</span><span>,
</span><span> </span><span style="color:#d69d85;">"Action"</span><span>: [</span><span style="color:#d69d85;">"s3:GetObject"</span><span>, </span><span style="color:#d69d85;">"s3:ListBucket"</span><span>],
</span><span> </span><span style="color:#d69d85;">"Resource"</span><span>: [</span><span style="color:#d69d85;">"arn:aws:s3:::my-research-data/*"</span><span>]
</span><span> }]
</span><span>}
</span></code></pre>
<p>This policy allows reading objects from a specific S3 bucket, nothing more. That's the <strong>principle of least privilege</strong>: grant only the minimum permissions needed to do the job.</p>
<p><strong>Why roles matter more than you think:</strong></p>
<p>The biggest mistake beginners make is hardcoding AWS credentials in their applications. I should be honest with you here, yes I've done it when I was younger. But today, please, I'm telling you, don't do this.</p>
<p><strong>Example scenario</strong>: Your EC2 instance needs to read files from S3.</p>
<p><strong>Wrong approach</strong>: Store AWS access keys in a config file on the instance.</p>
<ul>
<li>Keys can be stolen if the instance is compromised</li>
<li>Keys don't rotate automatically</li>
<li>Difficult to audit who used which credentials</li>
</ul>
<p><strong>Right approach</strong>: Attach an IAM role with S3 read permissions to the EC2 instance.</p>
<ul>
<li>No credentials stored anywhere</li>
<li>Temporary credentials rotate automatically</li>
<li>CloudTrail logs show exactly which instance accessed which resources</li>
</ul>
<p><strong>Root account protection:</strong></p>
<p>When you create an AWS account, you get a root user with unlimited access. This is dangerous. Best practices:</p>
<ol>
<li><strong>Enable MFA</strong> (Multi-Factor Authentication) on the root account</li>
<li><strong>Create IAM users</strong> for daily work, even for administrators</li>
<li><strong>Lock away root credentials</strong> - only use them for tasks that <em>require</em> root (billing, account closure)</li>
</ol>
<p><strong><a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html">AWS Organizations and Service Control Policies (SCPs)</a></strong> add another layer: they set permission boundaries across multiple accounts. Even if an IAM policy allows an action, an SCP can block it at the organization level. Useful for enforcing compliance (e.g., "no one can launch instances outside eu-west-3").</p>
<p><strong>In the biomedical research context:</strong></p>
<p>You'd use IAM to:</p>
<ul>
<li>Give researchers read-only access to specific S3 buckets containing datasets</li>
<li>Allow the data engineering team to manage ETL pipelines (write access to certain resources)</li>
<li>Grant your analysis scripts (running on EC2/Lambda) temporary credentials to access databases</li>
<li>Prevent anyone from accidentally exposing patient data by restricting public S3 bucket creation</li>
</ul>
<p>Security Groups (covered below) control <em>network</em> access. IAM controls <em>identity and authorization</em>. You need both.</p>
<h3 id="network">Network</h3>
<p><em>Maybe I did need this class after all. I'm having flashbacks... crimping tool... crossover cable... straight cable... cat5.... class C... IPV6... Layer 3... Layer 4....</em> Heeelp! \o/</p>
<p><strong>A <a href="https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html">VPC (Virtual Private Cloud)</a></strong> is your isolated network within AWS. With a VPC you define:</p>
<ul>
<li><strong><a href="https://aws.amazon.com/what-is/cidr/">CIDR block</a></strong>: your IP range (e.g., <code>10.0.0.0/16</code> gives you 65,536 IPs)</li>
<li><strong><a href="https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html">Subnets</a></strong>: subdivisions of your VPC, placed in specific Zones
<ul>
<li><em>Public subnets</em>: resources can have public IPs, direct internet access</li>
<li><em>Private subnets</em>: no direct internet access, only internal communication</li>
</ul>
</li>
</ul>
<p><strong>Gateways:</strong></p>
<ul>
<li><strong><a href="https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html">Internet Gateway (IGW)</a></strong>: enables internet access for public subnets. Attach one to your VPC, add routes, done.</li>
<li><strong><a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html">NAT Gateway</a></strong>: lets private subnet resources reach the internet (for updates, API calls) without being directly reachable..</li>
</ul>
<p><strong>Hybrid connectivity:</strong></p>
<ul>
<li><strong><a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html">Site-to-Site VPN</a></strong>: encrypted tunnel between your on-premises network and AWS VPC. Quick to set up, uses public internet, variable latency.</li>
<li><strong><a href="https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html">AWS Direct Connect</a></strong>: dedicated physical connection to AWS. Consistent latency, higher bandwidth, but requires physical setup and contracts. Useful for heavy data transfer or strict latency requirements.</li>
</ul>
<p><strong>Network Security:</strong></p>
<p>AWS provides two complementary layers of network security: <strong>Security Groups</strong> and <strong>Network ACLs</strong>. Understanding when to use each is critical.</p>
<p><strong><a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html">Security Groups</a></strong> are stateful firewalls attached to resources (EC2 instances, RDS databases, etc.). They act like bouncers at the door of individual servers. Define allowed inbound/outbound traffic by port, protocol, and source. If you allow inbound traffic on port 80, the return traffic is automatically allowed—that's what "stateful" means.</p>
<p><strong><a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html">Network ACLs (NACLs)</a></strong> are stateless firewalls at the subnet level. They filter traffic entering or leaving entire subnets. Rules are evaluated in numerical order (100, 200, 300...), and the first match wins. Unlike Security Groups, NACLs require explicit rules for both inbound and outbound traffic.</p>
<p><strong>Security Groups vs. Network ACLs:</strong></p>
<table><thead><tr><th>Feature</th><th>Security Groups</th><th>Network ACLs</th></tr></thead><tbody>
<tr><td><strong>Scope</strong></td><td>Instance level (attached to ENI)</td><td>Subnet level</td></tr>
<tr><td><strong>State</strong></td><td>Stateful (return traffic auto-allowed)</td><td>Stateless (must allow both directions)</td></tr>
<tr><td><strong>Rules</strong></td><td>Only Allow rules</td><td>Allow + Deny rules</td></tr>
<tr><td><strong>Evaluation</strong></td><td>All rules evaluated</td><td>Processed in order, first match wins</td></tr>
<tr><td><strong>Default behavior</strong></td><td>Deny all inbound, allow all outbound</td><td>Allow everything</td></tr>
<tr><td><strong>Typical use</strong></td><td>Your primary security control</td><td>Edge cases, compliance, IP blocking</td></tr>
</tbody></table>
<p><strong>When to use what:</strong></p>
<p><strong>Security Groups</strong> should be your default choice for 95% of scenarios:</p>
<ul>
<li>Allow SSH (port 22) only from your office IP</li>
<li>Let web servers accept HTTP/HTTPS from anywhere</li>
<li>Allow database connections only from application servers</li>
</ul>
<p><strong>Network ACLs</strong> are for specific edge cases:</p>
<ul>
<li>Block a known malicious IP range at the subnet boundary</li>
<li>Compliance requirements mandating subnet-level controls</li>
<li>Defense-in-depth: add a second layer of protection</li>
</ul>
<p><strong>Example scenario - Blocking an attacking IP:</strong></p>
<p>Your web servers (in a public subnet) are under attack from <code>203.0.113.0/24</code>. Security Groups can only <em>allow</em> traffic, not deny it. Solution: add a NACL rule.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>NACL Inbound Rules (evaluated in order):
</span><span>Rule 100: DENY TCP 80 from 203.0.113.0/24 (block attackers)
</span><span>Rule 200: ALLOW TCP 80 from 0.0.0.0/0 (allow everyone else)
</span><span>Rule *: DENY ALL ALL from 0.0.0.0/0 (default deny)
</span></code></pre>
<p>Because NACLs are stateless, you also need outbound rules for responses:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>NACL Outbound Rules:
</span><span>Rule 100: ALLOW TCP 1024-65535 to 0.0.0.0/0 (ephemeral ports)
</span><span>Rule *: DENY ALL ALL to 0.0.0.0/0 (default deny)
</span></code></pre>
<p>The <code>1024-65535</code> range covers ephemeral ports used by client connections. Yes, this is annoying. This is why most people stick to Security Groups.</p>
<p><strong>Pro tip:</strong> The default NACL in your VPC allows all traffic. Most AWS users never modify it. Only touch NACLs when you have <em>3</em> specific reasons.</p>
<p><strong><a href="https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html">Flow Logs</a></strong>: Capture network traffic metadata (source/destination IP, port, protocol, action) for debugging and audit. Essential for troubleshooting connectivity issues and security investigations. Send logs to CloudWatch or S3 for analysis.</p>
<h3 id="load-balancing">Load Balancing</h3>
<p><strong>High Availability</strong> means your application stays up even when parts of the infrastructure fail. In AWS, this typically involves deploying across multiple Availability Zones and using load balancers to distribute traffic.</p>
<p><strong><a href="https://aws.amazon.com/elasticloadbalancing/">Elastic Load Balancing (ELB)</a></strong> automatically distributes incoming traffic across multiple targets. AWS offers several types:</p>
<table><thead><tr><th>Type</th><th>Layer</th><th>Best For</th></tr></thead><tbody>
<tr><td><strong><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html">Application Load Balancer (ALB)</a></strong></td><td>Layer 7 (HTTP/HTTPS)</td><td>Web apps, microservices, content-based routing</td></tr>
<tr><td><strong><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">Network Load Balancer (NLB)</a></strong></td><td>Layer 4 (TCP/UDP)</td><td>Ultra-low latency, millions of requests/sec</td></tr>
<tr><td><strong><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/introduction.html">Gateway Load Balancer</a></strong></td><td>Layer 3</td><td>Third-party virtual appliances (firewalls, IDS)</td></tr>
</tbody></table>
<p><strong><a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html">Auto Scaling</a></strong> automatically adjusts the number of EC2 instances based on demand:</p>
<ul>
<li><strong>Scheduled scaling</strong>: scale at predictable times (e.g., business hours)</li>
<li><strong>Dynamic scaling</strong>: react to CloudWatch metrics (CPU, memory, custom metrics)</li>
<li><strong>Predictive scaling</strong>: ML-based forecasting of future demand</li>
</ul>
<p><strong>Multi-AZ resilience</strong>: Deploy instances across multiple AZs, place a load balancer in front, and Auto Scaling replaces failed instances automatically. This is the foundation of most production architectures on AWS.</p>
<h3 id="compute-services">Compute Services</h3>
<p><strong>Compute</strong> refers to the processing power needed to run applications. AWS offers several options depending on your level of control vs. convenience:</p>
<p><strong><a href="https://aws.amazon.com/ec2/">EC2 (Elastic Compute Cloud)</a></strong> — Virtual machines you fully control. Choose your OS, instance type (CPU/RAM), and storage. Pricing models:</p>
<table><thead><tr><th>Model</th><th>Description</th><th>Savings</th></tr></thead><tbody>
<tr><td><strong>On-Demand</strong></td><td>Pay by the hour/second, no commitment</td><td>Baseline pricing</td></tr>
<tr><td><strong>Reserved</strong></td><td>1 or 3-year commitment for steady workloads</td><td>Up to 72% off</td></tr>
<tr><td><strong>Spot</strong></td><td>Bid on unused capacity, can be interrupted</td><td>Up to 90% off</td></tr>
<tr><td><strong>Savings Plans</strong></td><td>Flexible commitment across instance families</td><td>Up to 72% off</td></tr>
</tbody></table>
<p><strong>Container services</strong> — For containerized workloads:</p>
<ul>
<li><strong><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html">ECS (Elastic Container Service)</a></strong>: AWS-native container orchestration. Simpler than Kubernetes, tightly integrated with AWS.</li>
<li><strong><a href="https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html">EKS (Elastic Kubernetes Service)</a></strong>: Managed Kubernetes. Use this if you need Kubernetes compatibility or already use K8s.</li>
<li><strong><a href="https://aws.amazon.com/fargate/">Fargate</a></strong>: Serverless compute for containers. No EC2 instances to manage—just define CPU/memory and run your containers.</li>
</ul>
<p><strong>When to use what:</strong></p>
<ul>
<li>EC2: Full control needed, legacy apps, specific OS requirements</li>
<li>ECS + Fargate: Simple containerized apps, AWS-native approach</li>
<li>EKS: Kubernetes expertise on the team, multi-cloud portability needed</li>
</ul>
<br>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/cloud-computing.jpg" alt="Cloud computing" title="Cloud computing"/>
<h3 id="storage">Storage</h3>
<p>AWS offers different storage types for different use cases:</p>
<p><strong><a href="https://aws.amazon.com/s3/">Amazon S3</a></strong> — Object storage for any type of data. Highly durable (11 nines), infinitely scalable. <a href="https://aws.amazon.com/s3/storage-classes/">Storage classes</a> optimize cost based on access patterns:</p>
<table><thead><tr><th>Class</th><th>Access Pattern</th><th>Retrieval</th></tr></thead><tbody>
<tr><td><strong>S3 Standard</strong></td><td>Frequent access</td><td>Instant</td></tr>
<tr><td><strong>S3 Intelligent-Tiering</strong></td><td>Unknown/changing patterns</td><td>Instant (auto-moves data)</td></tr>
<tr><td><strong>S3 Standard-IA</strong></td><td>Infrequent access</td><td>Instant</td></tr>
<tr><td><strong><a href="https://aws.amazon.com/s3/storage-classes/glacier/">S3 Glacier Instant</a></strong></td><td>Archive, rare access</td><td>Milliseconds</td></tr>
<tr><td><strong>S3 Glacier Flexible</strong></td><td>Archive</td><td>Minutes to hours</td></tr>
<tr><td><strong>S3 Glacier Deep Archive</strong></td><td>Long-term archive</td><td>12-48 hours</td></tr>
</tbody></table>
<p><strong>Lifecycle policies</strong> automatically transition objects between classes or delete them after a period.</p>
<p><strong>Block vs. File storage:</strong></p>
<ul>
<li><strong><a href="https://aws.amazon.com/ebs/">EBS (Elastic Block Store)</a></strong>: Block storage attached to a single EC2 instance. Think: hard drive for your VM. Great for databases.</li>
<li><strong><a href="https://aws.amazon.com/efs/">EFS (Elastic File System)</a></strong>: Shared file system mountable by multiple EC2 instances. Think: NFS. Great for shared application data.</li>
</ul>
<p><strong>Databases:</strong></p>
<p><strong><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html">Amazon RDS (Relational Database Service)</a></strong> — Managed relational databases for PostgreSQL, <em>(and other database providers I won't talk about because I am a free software advocate)</em>. AWS handles the operational heavy lifting: backups, patching, OS updates, and replication. You focus on schema design and queries.</p>
<p><strong>High Availability and scaling options:</strong></p>
<p>Understanding the difference between <strong>Multi-AZ</strong> and <strong>Read Replicas</strong> is critical:</p>
<table><thead><tr><th>Feature</th><th>Multi-AZ</th><th>Read Replicas</th></tr></thead><tbody>
<tr><td><strong>Purpose</strong></td><td>High availability (failover)</td><td>Read scaling, analytics</td></tr>
<tr><td><strong>Replication</strong></td><td>Synchronous (to standby in another AZ)</td><td>Asynchronous (can lag slightly)</td></tr>
<tr><td><strong>Failover</strong></td><td>Automatic (1-2 minutes)</td><td>Manual promotion required</td></tr>
<tr><td><strong>Readable</strong></td><td>Standby is not accessible</td><td>Yes, handle read traffic</td></tr>
<tr><td><strong>Use case</strong></td><td>Production databases requiring uptime</td><td>Distribute read load, reporting queries</td></tr>
<tr><td><strong>Cost</strong></td><td>~2x (paying for standby)</td><td>Additional instance cost per replica</td></tr>
</tbody></table>
<p><strong>Multi-AZ deployment</strong>: Your primary database synchronously replicates to a standby instance in a different Availability Zone. If the primary fails (hardware issue, AZ outage), RDS automatically fails over to the standby. Your application reconnects to the same endpoint—no code changes needed. This is for <strong>disaster recovery</strong>, not performance.</p>
<p><strong>Read Replicas</strong>: Create up to 15 read-only copies of your database. Use them to offload read traffic (analytics, reporting) from the primary. Replication is asynchronous, so there may be a slight lag (usually milliseconds to seconds). You can promote a replica to primary if needed, but it's a manual operation.</p>
<p><strong><a href="https://aws.amazon.com/rds/aurora/">Amazon Aurora</a></strong> — AWS's proprietary database engine, PostgreSQL compatible:</p>
<ul>
<li><strong>Performance</strong>: Up to 3x faster than PostgreSQL (according to AWS benchmarks)</li>
<li><strong>Storage auto-scaling</strong>: Starts at 10GB, grows automatically up to 128TB in 10GB increments</li>
<li><strong>High availability built-in</strong>: Data replicated 6 ways across 3 AZs automatically</li>
<li><strong>Read scaling</strong>: Up to 15 Aurora Replicas with sub-10ms replica lag</li>
<li><strong>Cost</strong>: More expensive than standard RDS, but better price-to-performance for demanding workloads</li>
</ul>
<p><a href="https://www.cloudzero.com/blog/aurora-vs-rds/">Aurora</a> is AWS's recommended choice for new applications that need high performance and availability.</p>
<p><strong>Backup Strategies:</strong></p>
<p>RDS provides two backup mechanisms:</p>
<ol>
<li>
<p><strong><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html">Automated backups</a></strong>:</p>
<ul>
<li>Enabled by default</li>
<li>Point-in-time recovery: restore your database to any second within the retention period (1-35 days)</li>
<li>Full daily backups + transaction logs</li>
<li>Deleted when you delete the RDS instance (unless you configure a final snapshot)</li>
</ul>
</li>
<li>
<p><strong><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html">Manual snapshots</a></strong>:</p>
<ul>
<li>User-initiated, persist until you explicitly delete them</li>
<li>Useful before major changes (schema migrations, upgrades)</li>
<li>Can be copied across regions for disaster recovery</li>
</ul>
</li>
</ol>
<p><strong>Pro tip</strong>: Before any risky operation (major version upgrade, schema change), create a manual snapshot. Automated backups might not cover you if the retention period expires.</p>
<p><strong>RDS vs. NoSQL:</strong></p>
<p>Not all data fits the relational model. <strong><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html">Amazon DynamoDB</a></strong> is AWS's managed NoSQL database (key-value and document store):</p>
<table><thead><tr><th>Feature</th><th>RDS (SQL)</th><th>DynamoDB (NoSQL)</th></tr></thead><tbody>
<tr><td><strong>Data model</strong></td><td>Tables with fixed schema, relations</td><td>Key-value / JSON documents, flexible schema</td></tr>
<tr><td><strong>Queries</strong></td><td>Complex SQL (JOINs, aggregations)</td><td>Simple key lookups, limited querying</td></tr>
<tr><td><strong>Scaling</strong></td><td>Vertical (bigger instance), Read Replicas</td><td>Horizontal (automatic), virtually unlimited</td></tr>
<tr><td><strong>Transactions</strong></td><td>Full ACID support</td><td>Limited transactions (single-item ACID, multi-item with conditions)</td></tr>
<tr><td><strong>Best for</strong></td><td>Complex queries, reporting, traditional apps</td><td>High-throughput simple queries, IoT, gaming, mobile backends</td></tr>
<tr><td><strong>Pricing</strong></td><td>Pay for instance size (even if idle)</td><td>Pay for storage + read/write capacity used</td></tr>
</tbody></table>
<p><strong>When to use what:</strong></p>
<ul>
<li><strong>RDS</strong>: You need SQL, complex queries, transactions, existing relational schema, or you're migrating from on-premises MySQL/PostgreSQL</li>
<li><strong>DynamoDB</strong>: Key-value access patterns, need massive scale, unpredictable traffic spikes, single-digit millisecond latency requirements</li>
</ul>
<p>Most traditional applications (e-commerce, ERP, content management) use RDS. DynamoDB shines for high-scale, low-latency use cases where you don't need complex querying.</p>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/cloud-storage.jpg" alt="cloud storage" title="cloud storage"/>
<h3 id="observability">Observability</h3>
<p>Observability = understanding what's happening inside your systems. AWS provides <strong><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html">Amazon CloudWatch</a></strong> as the central service for this.</p>
<p><strong><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html">Metrics</a></strong>: Numerical data points over time. AWS services automatically send metrics (CPU usage, network traffic, request counts). You can also publish custom metrics from your applications.</p>
<p><strong><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html">Logs</a></strong>: Centralized log storage and analysis. Stream logs from EC2, Lambda, containers, or any application. Use <strong>Log Insights</strong> to query across log groups with a SQL-like syntax.</p>
<p><strong><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html">Alarms</a></strong>: Watch metrics and trigger actions when thresholds are breached:</p>
<ul>
<li>Send notifications via SNS (email, SMS, Slack)</li>
<li>Trigger Auto Scaling to add/remove instances</li>
<li>Execute Lambda functions for custom remediation</li>
</ul>
<p><strong>Example alarm</strong>: "If average CPU > 80% for 5 minutes, add 2 instances and notify the ops team."</p>
<p><strong>Beyond CloudWatch:</strong></p>
<ul>
<li><strong><a href="https://aws.amazon.com/xray/">AWS X-Ray</a></strong>: Distributed tracing for microservices</li>
<li><strong><a href="https://aws.amazon.com/cloudtrail/">CloudTrail</a></strong>: Audit log of all API calls in your account</li>
</ul>
<br>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/cloud-observability.jpg" alt="cloud observability" title="cloud observability"/>
<br>
<h2 id="one-picture-is-worth-thousand-words">One picture is worth thousand words</h2>
<div class="aws-diagram-wrapper">
<style>
.aws-diagram-wrapper {
--aws-bg: #0d1117;
--aws-bg-secondary: #161b22;
--aws-bg-tertiary: #21262d;
--aws-border: #30363d;
--aws-text: #e6edf3;
--aws-text-muted: #8b949e;
--aws-orange: #ff9900;
--aws-green: #238636;
--aws-blue: #1f6feb;
--aws-blue-light: #388bfd;
--aws-purple: #a371f7;
--aws-yellow: #f0883e;
font-family: 'IBM Plex Sans', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
background: var(--aws-bg);
color: var(--aws-text);
padding: 40px 20px;
border-radius: 8px;
margin: 30px 0;
}
.aws-diagram-wrapper * {
margin: 0;
padding: 0;
box-sizing: border-box;
}
.aws-diagram-wrapper h1 {
font-size: 1.6rem;
font-weight: 600;
margin-bottom: 8px;
color: var(--aws-orange);
}
.aws-diagram-wrapper .subtitle {
color: var(--aws-text-muted);
font-size: 0.9rem;
margin-bottom: 40px;
}
.aws-diagram-wrapper .diagram-container {
position: relative;
background: linear-gradient(180deg, var(--aws-bg-secondary) 0%, var(--aws-bg) 100%);
border: 1px solid var(--aws-border);
border-radius: 8px;
padding: 30px;
overflow-x: auto;
}
.aws-diagram-wrapper .diagram {
display: flex;
flex-direction: column;
gap: 16px;
min-width: 1100px;
}
.aws-diagram-wrapper .zone {
border: 2px dashed var(--aws-border);
border-radius: 8px;
padding: 20px;
position: relative;
}
.aws-diagram-wrapper .zone-label {
position: absolute;
top: -10px;
left: 20px;
background: var(--aws-bg);
padding: 2px 12px;
font-size: 0.7rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 1px;
color: var(--aws-text-muted);
}
.aws-diagram-wrapper .zone.internet {
border-color: var(--aws-green);
background: rgba(35, 134, 54, 0.05);
}
.aws-diagram-wrapper .zone.vpc {
border-color: var(--aws-blue);
background: rgba(31, 111, 235, 0.03);
}
.aws-diagram-wrapper .zone.public-subnet {
border-color: var(--aws-yellow);
background: rgba(240, 136, 62, 0.05);
}
.aws-diagram-wrapper .zone.private-subnet {
border-color: var(--aws-purple);
background: rgba(163, 113, 247, 0.05);
}
.aws-diagram-wrapper .zone.az {
border-color: var(--aws-blue-light);
border-style: dotted;
background: rgba(56, 139, 253, 0.03);
}
.aws-diagram-wrapper .row {
display: flex;
gap: 20px;
align-items: flex-start;
}
.aws-diagram-wrapper .component {
background: var(--aws-bg-tertiary);
border: 1px solid var(--aws-border);
border-radius: 6px;
padding: 14px 18px;
min-width: 130px;
text-align: center;
position: relative;
transition: all 0.2s ease;
cursor: pointer;
}
.aws-diagram-wrapper .component:hover {
border-color: var(--aws-orange);
transform: translateY(-2px);
box-shadow: 0 4px 20px rgba(255, 153, 0, 0.15);
}
.aws-diagram-wrapper .component.active {
border-color: var(--aws-orange);
background: #2d333b;
}
.aws-diagram-wrapper .component-name {
font-weight: 600;
font-size: 0.85rem;
margin-bottom: 6px;
color: var(--aws-text);
}
.aws-diagram-wrapper .component-type {
font-family: 'JetBrains Mono', 'Fira Code', monospace;
font-size: 0.65rem;
color: var(--aws-orange);
background: rgba(255, 153, 0, 0.1);
padding: 3px 8px;
border-radius: 3px;
display: inline-block;
}
.aws-diagram-wrapper .arrow-container {
display: flex;
align-items: center;
justify-content: center;
padding: 8px 0;
}
.aws-diagram-wrapper .arrow {
display: flex;
align-items: center;
gap: 8px;
color: #6e7681;
font-size: 0.7rem;
font-family: 'JetBrains Mono', monospace;
}
.aws-diagram-wrapper .arrow-line {
width: 50px;
height: 2px;
background: linear-gradient(90deg, var(--aws-border) 0%, var(--aws-orange) 50%, var(--aws-border) 100%);
position: relative;
}
.aws-diagram-wrapper .arrow-line::after {
content: '>';
position: absolute;
right: -6px;
top: -7px;
color: var(--aws-orange);
font-size: 0.7rem;
font-weight: bold;
}
.aws-diagram-wrapper .arrow-down {
display: flex;
flex-direction: column;
align-items: center;
}
.aws-diagram-wrapper .arrow-down .arrow-line {
width: 2px;
height: 30px;
background: linear-gradient(180deg, var(--aws-border) 0%, var(--aws-orange) 50%, var(--aws-border) 100%);
}
.aws-diagram-wrapper .arrow-down .arrow-line::after {
content: 'v';
right: -5px;
top: auto;
bottom: -10px;
font-size: 0.6rem;
}
.aws-diagram-wrapper .flex-center {
display: flex;
justify-content: center;
align-items: center;
}
.aws-diagram-wrapper .flex-between {
display: flex;
justify-content: space-between;
align-items: flex-start;
}
.aws-diagram-wrapper .flex-col {
display: flex;
flex-direction: column;
align-items: center;
gap: 8px;
}
.aws-diagram-wrapper .info-panel {
margin-top: 30px;
background: var(--aws-bg-secondary);
border: 1px solid var(--aws-border);
border-radius: 8px;
padding: 24px;
min-height: 180px;
}
.aws-diagram-wrapper .info-panel h3 {
font-size: 1rem;
color: var(--aws-orange);
margin-bottom: 12px;
}
.aws-diagram-wrapper .info-panel .description {
color: var(--aws-text-muted);
line-height: 1.7;
margin-bottom: 16px;
font-size: 0.9rem;
}
.aws-diagram-wrapper .info-panel .analogy {
background: var(--aws-bg-tertiary);
border-left: 3px solid var(--aws-green);
padding: 12px 16px;
border-radius: 0 6px 6px 0;
margin-bottom: 16px;
}
.aws-diagram-wrapper .info-panel .analogy-label {
font-size: 0.65rem;
text-transform: uppercase;
letter-spacing: 1px;
color: var(--aws-green);
margin-bottom: 6px;
}
.aws-diagram-wrapper .info-panel .analogy p {
font-size: 0.85rem;
color: var(--aws-text-muted);
}
.aws-diagram-wrapper .info-panel .code {
font-family: 'JetBrains Mono', 'Fira Code', monospace;
font-size: 0.75rem;
background: var(--aws-bg);
padding: 12px 16px;
border-radius: 6px;
color: #79c0ff;
overflow-x: auto;
line-height: 1.5;
}
.aws-diagram-wrapper .storage-section {
display: flex;
gap: 20px;
margin-top: 16px;
}
.aws-diagram-wrapper .storage-group {
flex: 1;
background: var(--aws-bg-secondary);
border: 1px solid var(--aws-border);
border-radius: 6px;
padding: 16px;
}
.aws-diagram-wrapper .storage-group h4 {
font-size: 0.75rem;
color: var(--aws-text-muted);
margin-bottom: 12px;
text-transform: uppercase;
letter-spacing: 1px;
}
.aws-diagram-wrapper .storage-items {
display: flex;
gap: 12px;
flex-wrap: wrap;
}
.aws-diagram-wrapper .legend {
display: flex;
gap: 24px;
flex-wrap: wrap;
margin-top: 20px;
padding: 14px;
background: var(--aws-bg-secondary);
border-radius: 6px;
}
.aws-diagram-wrapper .legend-item {
display: flex;
align-items: center;
gap: 8px;
font-size: 0.75rem;
color: var(--aws-text-muted);
}
.aws-diagram-wrapper .legend-color {
width: 14px;
height: 14px;
border-radius: 3px;
border: 2px dashed;
}
.aws-diagram-wrapper .legend-color.internet { border-color: var(--aws-green); background: rgba(35, 134, 54, 0.2); }
.aws-diagram-wrapper .legend-color.vpc { border-color: var(--aws-blue); background: rgba(31, 111, 235, 0.2); }
.aws-diagram-wrapper .legend-color.public { border-color: var(--aws-yellow); background: rgba(240, 136, 62, 0.2); }
.aws-diagram-wrapper .legend-color.private { border-color: var(--aws-purple); background: rgba(163, 113, 247, 0.2); }
.aws-diagram-wrapper .click-hint {
text-align: center;
color: #6e7681;
font-size: 0.75rem;
margin-top: 14px;
}
.aws-diagram-wrapper .or-divider {
font-size: 0.65rem;
color: #6e7681;
font-style: italic;
}
</style>
<h1>AWS Architecture - Overview</h1>
<p class="subtitle">Click on a component to see details and analogy with your current stack</p>
<div class="diagram-container">
<div class="diagram">
<!-- Internet Zone -->
<div class="zone internet">
<span class="zone-label">Internet</span>
<div class="flex-center">
<div class="component" data-component="users">
<div class="component-name">Users</div>
<div class="component-type">Front & mobile Apps</div>
</div>
<div class="arrow">
<div class="arrow-line"></div>
<span>HTTPS</span>
<div class="arrow-line"></div>
</div>
<div class="component" data-component="route53">
<div class="component-name">Route 53</div>
<div class="component-type">DNS</div>
</div>
</div>
</div>
<!-- VPC -->
<div class="zone vpc">
<span class="zone-label">VPC (10.0.0.0/16)</span>
<div class="flex-between" style="gap: 20px;">
<!-- AZ 1 -->
<div class="zone az" style="flex: 1;">
<span class="zone-label">AZ-1 (eu-west-3a)</span>
<!-- Public Subnet AZ1 -->
<div class="zone public-subnet" style="margin-bottom: 15px;">
<span class="zone-label">Public Subnet (10.0.1.0/24)</span>
<div class="flex-col">
<div class="component" data-component="igw">
<div class="component-name">Internet Gateway</div>
<div class="component-type">IGW</div>
</div>
<div class="arrow-down"><div class="arrow-line"></div></div>
<div class="component" data-component="alb">
<div class="component-name">Load Balancer</div>
<div class="component-type">ALB / NLB</div>
</div>
<div class="component" data-component="nat">
<div class="component-name">NAT Gateway</div>
<div class="component-type">NAT</div>
</div>
</div>
</div>
<!-- Private Subnet AZ1 -->
<div class="zone private-subnet">
<span class="zone-label">Private Subnet (10.0.10.0/24)</span>
<div class="flex-col">
<div class="component" data-component="ec2">
<div class="component-name">EC2 Instance</div>
<div class="component-type">t3.medium</div>
</div>
<div class="or-divider">or</div>
<div class="component" data-component="ecs">
<div class="component-name">ECS / Fargate</div>
<div class="component-type">Container</div>
</div>
</div>
</div>
</div>
<!-- AZ 2 -->
<div class="zone az" style="flex: 1;">
<span class="zone-label">AZ-2 (eu-west-3b)</span>
<!-- Public Subnet AZ2 -->
<div class="zone public-subnet" style="margin-bottom: 15px;">
<span class="zone-label">Public Subnet (10.0.2.0/24)</span>
<div class="flex-col">
<div class="component" data-component="alb">
<div class="component-name">Load Balancer</div>
<div class="component-type">ALB (replica)</div>
</div>
</div>
</div>
<!-- Private Subnet AZ2 -->
<div class="zone private-subnet">
<span class="zone-label">Private Subnet (10.0.20.0/24)</span>
<div class="flex-col">
<div class="component" data-component="ec2">
<div class="component-name">EC2 Instance</div>
<div class="component-type">t3.medium</div>
</div>
<div class="arrow-down"><div class="arrow-line"></div></div>
<div class="component" data-component="rds">
<div class="component-name">RDS PostgreSQL</div>
<div class="component-type">Multi-AZ</div>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Storage & Monitoring -->
<div class="storage-section">
<div class="storage-group">
<h4>Storage</h4>
<div class="storage-items">
<div class="component" data-component="s3">
<div class="component-name">S3 Bucket</div>
<div class="component-type">Object Storage</div>
</div>
<div class="component" data-component="ebs">
<div class="component-name">EBS Volume</div>
<div class="component-type">Block Storage</div>
</div>
<div class="component" data-component="efs">
<div class="component-name">EFS</div>
<div class="component-type">File System</div>
</div>
</div>
</div>
<div class="storage-group">
<h4>Monitoring and Streaming</h4>
<div class="storage-items">
<div class="component" data-component="cloudwatch">
<div class="component-name">CloudWatch</div>
<div class="component-type">Monitoring</div>
</div>
<div class="component" data-component="msk">
<div class="component-name">MSK (Kafka)</div>
<div class="component-type">Streaming</div>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Legend -->
<div class="legend">
<div class="legend-item">
<div class="legend-color internet"></div>
<span>Internet (public access)</span>
</div>
<div class="legend-item">
<div class="legend-color vpc"></div>
<span>VPC (your private network)</span>
</div>
<div class="legend-item">
<div class="legend-color public"></div>
<span>Public Subnet (exposed)</span>
</div>
<div class="legend-item">
<div class="legend-color private"></div>
<span>Private Subnet (protected)</span>
</div>
</div>
<p class="click-hint">Click on a component to see details</p>
<!-- Info Panel -->
<div class="info-panel" id="aws-info-panel">
<h3>Select a component</h3>
<p class="description">
Click on any element of the diagram to see a detailed explanation
and the equivalent in your current environment (Nginx, Django, PostgreSQL...).
</p>
</div>
</div>
<script>
(function() {
const componentInfo = {
users: {
title: "Users / Clients",
description: "The users who access your applications from the Internet. This could be health workers using our <a href='https://nskm.xyz/posts/syndromic-surveillance/'>syndromic surveillance system</a>.",
analogy: "It's the same as now - your users typing the URL in their browser or systems calling your REST API.",
code: "curl https://surveillance.pasteur.sn/api/v1/cases/"
},
route53: {
title: "Route 53 - DNS Service",
description: "AWS's managed DNS service. It translates domain names (surveillance.pasteur.sn) into IP addresses. Can perform health checking and intelligent routing (geolocation, failover).",
analogy: "Equivalent to your current DNS (maybe managed by your registrar or Cloudflare). The difference: Route 53 integrates natively with other AWS services.",
code: "# Example record<br>surveillance.pasteur.sn A -> ALB (alias)"
},
igw: {
title: "Internet Gateway (IGW)",
description: "The gateway in and out of your VPC to the Internet. Without IGW, nothing in your VPC can communicate with the outside world. There is ONE per VPC.",
analogy: "It's like the router/firewall of your datacenter that connects your internal network to the Internet. Like your platform's main gateway.",
code: "# One VPC = One IGW<br>aws ec2 create-internet-gateway"
},
alb: {
title: "Application Load Balancer (ALB / NLB)",
description: "Distributes traffic between multiple instances. ALB = Layer 7 (HTTP), understands URLs and can route /api to one group, /admin to another. NLB = Layer 4 (TCP), faster but less intelligent.",
analogy: "It's your Nginx in reverse proxy mode, but managed and automatically scalable. No need to manage the nginx.conf file anymore!",
code: "# Nginx equivalent<br>upstream django {<br> server 10.0.10.5:8000;<br> server 10.0.20.5:8000;<br>}<br>location / {<br> proxy_pass http://django;<br>}"
},
nat: {
title: "NAT Gateway",
description: "Allows resources in a PRIVATE subnet to access the Internet (for updates, external APIs) WITHOUT being accessible from the Internet. Outgoing traffic goes through NAT, but no incoming traffic is possible.",
analogy: "Like when your Django server (behind a firewall) can do 'pip install' or call the Africa's Talking API, but no one can access that server directly from the Internet.",
code: "# From your private EC2, this works:<br>curl https://api.africastalking.com/send<br><br># But from the Internet, impossible to reach the EC2"
},
ec2: {
title: "EC2 - Elastic Compute Cloud",
description: "Virtual machines. You choose the type (t3.medium = 2 vCPU, 4 GB RAM), the OS (Ubuntu, Amazon Linux), and you pay per hour. On Demand = flexible, Reserved = -60%, Spot = -90% but interruptible.",
analogy: "It's exactly like your current VPS or physical servers, but elastic. You can create 10 in 2 minutes, and delete them afterwards.",
code: "# Equivalent to your current server<br>Ubuntu 22.04<br>2 vCPU, 4 GB RAM<br>Django + Gunicorn + your code"
},
ecs: {
title: "ECS / EKS / Fargate - Containers",
description: "ECS = AWS container orchestrator. EKS = Managed Kubernetes. Fargate = serverless (no servers to manage). You package your Django in a Docker image, and AWS handles the rest.",
analogy: "If you're using Docker now with docker-compose, ECS is the same but in production and scalable. No need to SSH into the server to do 'docker pull'.",
code: "# Dockerfile<br>FROM python:3.11<br>COPY . /app<br>RUN pip install -r requirements.txt<br>CMD [\"gunicorn\", \"config.wsgi:application\"]"
},
rds: {
title: "RDS - Relational Database Service",
description: "Managed databases (PostgreSQL, MySQL, etc.). AWS handles automatic backups, security patches, Multi-AZ replication. You focus on your queries, not maintenance.",
analogy: "It's your current PostgreSQL, but you no longer have to worry about pg_dump, version upgrades, or replication. AWS does everything.",
code: "# Django settings.py<br>DATABASES = {<br> 'default': {<br> 'ENGINE': 'django.db.backends.postgresql',<br> 'HOST': 'mydb.xxxxx.eu-west-3.rds.amazonaws.com',<br> 'NAME': 'surveillance_db',<br> }<br>}"
},
s3: {
title: "S3 - Simple Storage Service",
description: "Unlimited object storage. Perfect for static files, backups, media uploads. Not a classic file system (no mkdir), but 'buckets' and 'objects' with URLs.",
analogy: "Replaces your /var/www/media/ folder for Django uploads. No more disk space issues, and your files are accessible via URL directly.",
code: "# Django settings with django-storages<br>DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'<br>AWS_STORAGE_BUCKET_NAME = 'ipd-surveillance-media'"
},
ebs: {
title: "EBS - Elastic Block Store",
description: "Virtual hard drives attached to an EC2 instance. Like your server's SSD. Persistent (data remains even if EC2 is stopped). Different types: gp3 (general), io2 (high perf), st1 (cheap HDD).",
analogy: "It's exactly your current server's hard drive. /dev/sda1 but in the cloud. If your EC2 dies, you can attach the EBS to a new instance.",
code: "# On your EC2<br>df -h<br>/dev/nvme0n1p1 100G 45G 55G 45% /"
},
efs: {
title: "EFS - Elastic File System",
description: "File system shared between MULTIPLE EC2 instances. Like a NAS. Useful when multiple servers need to access the same files (ex: shared uploads, common source code).",
analogy: "Like an NFS share between your servers. If you have 3 Django instances that all need to see the same uploaded files, EFS solves this problem.",
code: "# Mounted on multiple EC2<br>mount -t nfs4 fs-xxxxx.efs.eu-west-3.amazonaws.com:/ /mnt/shared"
},
cloudwatch: {
title: "CloudWatch - Monitoring",
description: "Collects metrics (CPU, RAM, requests), centralizes logs, and triggers alerts. Built-in dashboard. Can trigger automatic actions (ex: scale up if CPU > 80%).",
analogy: "Combines htop + tail -f /var/log + your alert system. But everything is centralized and you can create visual dashboards.",
code: "# Alert example<br>If CPUUtilization > 80% for 5 min<br>-> Send email to ops@pasteur.sn<br>-> Add 2 instances to Auto Scaling Group"
},
msk: {
title: "MSK - Managed Streaming for Kafka",
description: "Apache Kafka managed by AWS. For real-time data streaming, data pipelines, or event-driven architecture. Useful for processing millions of events per second.",
analogy: "Imagine each suspected case report generates an 'event' that must be processed in real-time to trigger epidemiological alerts. Kafka enables this continuous flow of data.",
code: "# Producer (notification system)<br>producer.send('alerts-topic', {<br> 'type': 'suspected_case',<br> 'disease': 'cholera',<br> 'location': 'dakar',<br> 'timestamp': '2025-01-31T10:30:00Z'<br>})"
}
};
const wrapper = document.querySelector('.aws-diagram-wrapper');
const panel = wrapper.querySelector('#aws-info-panel');
wrapper.querySelectorAll('.component').forEach(comp => {
comp.addEventListener('click', function() {
const componentId = this.dataset.component;
const info = componentInfo[componentId];
if (info) {
wrapper.querySelectorAll('.component').forEach(c => c.classList.remove('active'));
wrapper.querySelectorAll(`[data-component="${componentId}"]`).forEach(c => c.classList.add('active'));
panel.innerHTML = `
<h3>${info.title}</h3>
<p class="description">${info.description}</p>
<div class="analogy">
<div class="analogy-label">Equivalent dans ton stack actuel</div>
<p>${info.analogy}</p>
</div>
<div class="code">${info.code.replace(/\n/g, '<br>')}</div>
`;
}
});
});
})();
</script>
<h2 id="how-to-practice">How to practice</h2>
<p>You don't need a corporate account to learn AWS:</p>
<ul>
<li><strong><a href="https://aws.amazon.com/free/">AWS Free Tier</a></strong>: The real AWS, free for 12 months (with limits). Great for hands-on experience with actual services.</li>
<li><strong><a href="https://localstack.cloud/">LocalStack</a></strong>: AWS emulator running on your laptop. Spin up S3, Lambda, DynamoDB locally. No cloud costs.</li>
<li><strong><a href="https://minikube.sigs.k8s.io/">Minikube</a> / <a href="https://kind.sigs.k8s.io/">Kind</a></strong>: Local Kubernetes clusters for learning EKS concepts.</li>
<li><strong><a href="https://k3s.io/">K3s</a> / <a href="https://k3d.io/">K3d</a></strong>: Lightweight Kubernetes distributions, perfect for development and edge deployments.</li>
<li><strong><a href="https://docs.podman.io/en/latest/markdown/podman-compose.1.html">Podman Compose</a></strong>: Simulate architectures with containers (PostgreSQL, Redis, Nginx). Understand the concepts before going managed.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>The key insight is this: everything you've been doing manually for years, fingers in the nose, eyes closed (configuring Nginx, Gunicorn, PostgreSQL, managing users, monitoring logs), AWS offers it as managed services. You're trading full control for less maintenance.</p>
<p>Is that trade-off worth it? It depends. For a small team that needs to move fast and can't afford dedicated ops, managed services make sense. For organizations with specific compliance needs or those who want to avoid vendor lock-in, the calculus changes.</p>
<p>Either way, understanding these concepts matters, whether you end up using AWS, another cloud, or staying on-premises.</p>
<img style="display: block; margin: 0 auto; width: 400px" src="/images/there_is_no_cloud.png" alt="There is no cloud" title="There is no cloud"/>
<h2 id="more-on-this-topic">More on this topic</h2>
<p>Congratulations, you made it to the end. I hope you learned something, if you enjoyed this piece, you can read my post about <a href="https://nskm.xyz/posts/syndromic-surveillance/">how I'm helping save lives in West Africa using free software</a>, or you can <em>like, share and subscribe; it helps fight the algorithm.</em> Here are some links you may find interesing:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=uaflChh3n1A">Introduction to AWS</a></li>
<li><a href="https://xkcd.com/908/">The Cloud</a></li>
<li><a href="https://www.youtube.com/watch?v=MQbkN99eBD8">Why is Kubernetes everywhere?</a></li>
<li><a href="https://basecamp.com/cloud-exit">We left the cloud</a></li>
<li><a href="https://www.youtube.com/watch?v=vBjonut1JMk">How Kubernetes is built</a></li>
<li><a href="https://www.youtube.com/watch?v=tCehFMwjWqM">AI won't take your job (but this will)</a></li>
<li><a href="https://www.youtube.com/watch?v=CjKhQoYeR4Q">Getting started with AWS, step by step guide</a></li>
</ul>
Syndromic Surveillance! (Part 1)2025-08-24T00:00:00+00:002025-08-24T00:00:00+00:00
Unknown
https://nskm.xyz/posts/syndromic-surveillance/ <img style="display: block; margin: 0 auto; width: 600px" src="/images/surveillance_screens.jpg" alt="" title="Liberty"/>
<br>
<br>
<h2 id="disclaimers">Disclaimers :</h2>
<ol>
<li>
<p>Opinions expressed in this post <em>(and in any of all my posts)</em> are solely, <em>unless otherwise specified</em>, those of the authors, <em>me</em>. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups.</p>
</li>
<li>
<p>I've strived for accuracy throughout this piece, but if you catch any errors, please reach out—I'd be grateful for the feedback and happy to make updates!</p>
</li>
</ol>
<br>
<br>
<h2 id="toc">TOC:</h2>
<ul>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#hook">Hook</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#history">Definition, context, history</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#what">4S overview</a>
<ul>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#core">Core principles</a>
<ul>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#collect">Data collection</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#process">Data processing and storage</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#share">Data sharing</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#alert">Alerting and notifications</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#integration">Integration with existing systems</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#stack">The technological stack</a>
<ul>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#front">Frontend</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#back">Backend</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#else">More</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#engineers">The engineering team</a></li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#future">The roadmap</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/syndromic-surveillance/#more">More on this topic</a></li>
</ul>
<br>
<br>
<h2 id="hook"><u>Hook:</u></h2>
<p>From <strong>June 30 to July 2, 2025</strong>, a coordination meeting for the entire 4S network was recently held in <a href="https://en.wikipedia.org/wiki/Lom%C3%A9">Lomé, Togo</a>. During this meeting, a new implementation of the <strong>4S platform</strong> was presented to the participants.</p>
<video style="display: block; margin: 0 auto; width: 600px" class="video" controls="controls" >
<source src="/images/4s-annual-regional-coordination-meeting.mp4" type="video/mp4" />
Your browser does not support the video tag.
</video>
<p>Quote:</p>
<blockquote>
<p>Le réseau 4S a été conçu pour répondre à un besoin crucial, celui de disposer d'un système régional de surveillance capable de détecter précocément les maladies de santé publique, de renforcer les capacités de laboratoires et de partager des données pour une réponse adéquate et adaptée aux problèmes sanitaires.</p>
</blockquote>
<p>Quote :</p>
<blockquote>
<p>The 4S network was designed to meet a crucial need: to have a regional surveillance system capable of early detection of public health diseases, strengthening laboratory capacities and sharing data for an adequate and appropriate response to health issues.
<br>
<br></p>
</blockquote>
<p>The striking disconnect that exists across many sectors, public health included. Officials excel at announcing initiatives and securing funding. Press releases about <em>"implementing advanced early detection systems"</em> generate buzz and demonstrate progress. But when it comes to the unglamorous technical work: database design, API development, system integration... Silence. These crucial implementation details, handled by <em>anonymous engineers and developers</em>, rarely see daylight in public discourse.</p>
<p>While policy makers avoid technical specifics, it's precisely these software choices that determine whether systems function during critical moments.</p>
<br>
<img style="display: block; margin: 0 auto; width: 250px" src="/images/mother.jpg" alt="Mother" title="Mother"/>
<br>
<br>
<h2 id="history"><u>Definition, context, history:</u></h2>
<p>A syndromic surveillance system is a <a href="https://www.youtube.com/watch?v=Y6DPDC_Mf90">public health</a> <strong>surveillance</strong> system that monitors <strong>symptom clusters</strong> rather than specific confirmed diagnoses. It aims to <strong>detect epidemics or public health events</strong> at an <strong>early</strong> stage, by analyzing data in real time.</p>
<h3 id="2007">2007:</h3>
<p>Before 2007, <a href="https://simple.wikipedia.org/wiki/Madagascar">Madagascar</a> only had a passive, monthly paper-based reporting system that was <em><strong>too slow</strong></em> for outbreak response. The <em><a href="https://pubmed.ncbi.nlm.nih.gov/17505252/">chikungunya outbreak</a></em> in the Indian Ocean exposed <em><strong>critical gaps</strong></em> in Madagascar's disease surveillance.</p>
<p>The solution was to setup a real-time sentinel surveillance system using mobile phones for daily data transmission. This implementation was <em><strong>the first nationwide real-time-like</strong></em> <a href="https://pdfs.semanticscholar.org/66b8/8be85954b548c3b744ffd7ea8bac7ccae008.pdf">surveillance system</a> ever established in Madagascar.</p>
<h3 id="2012">2012:</h3>
<p>The Syndromic Sentinel Surveillance System in Senegal, the 4S network, is <a href="https://rh.pasteur.sn/en/research-and-public-health/epidemiology-clinical-research-and-data-science/fever-surveillance-syndromic-sentinel-surveillance-senegal-4s-network">Another syndromic sentinel surveillance system</a> based on febrile syndromes. It was implemented in Senegal in 2012 and inpired by the Madagascar model.</p>
<p>The criteria of individual target diseases are based on <a href="https://www.who.int/publications/i/item/who-recommended-surveillance-standards">World Health Organization recommended surveillance standards</a>. This means: for <em>each disease</em> or <em>syndrome</em> there is a description of the <em>rationale for surveillance, case definition, types of surveillance, minimum data elements, data analyses and principal uses of data for decision-making.</em></p>
<p>Data are transmitted on a daily basis by the general practitioners of the sentinel sites. They are validated and analyzed daily within the epidemiology unit of the Institution. These data are sent to the Ministry of Health in the form of a weekly newsletter.</p>
<h3 id="2015">2015:</h3>
<p>The existing implementation faced significant challenges. The database consisted of scattered Excel files distributed across multiple locations, making studies and analyses complicated and time-consuming. There was an urgent need to organize, structure, and centralize these disparate files.</p>
<p>To address these issues, a new platform was developed: Teranga. Built using <a href="https://httpd.apache.org/">Apache</a>, <a href="https://www.php.net/">PHP</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript">Javascript</a> and <a href="https://www.mysql.com/products/community/">MySQL</a>, Teranga provides improved data management, enhanced tracking capabilities, modular architecture, and streamlined deployment during outbreak situations.</p>
<p>The new sentinel surveillance system included 17 health centers and <a href="https://www.omicsonline.org/proceedings/early-outbreak-detection-through-sentinel-surveillance-system-in-senegal-51640.html">identified 4 confirmed outbreaks</a>. The system has proved the feasibility of improving disease surveillance capacity through innovative technological solutions.</p>
<h3 id="2020">2020:</h3>
<p>When <a href="https://www.cdc.gov/covid/about/index.html">COVID-19</a> struck, Teranga rose to meet an unprecedented challenge, becoming the backbone of Senegal's pandemic response. Seamlessly coordinating patient care, traveler monitoring, financial operations, clinical research, and treatment center management across the entire country. The pandemic trully elevated Teranga to a mission-critical status.</p>
<h3 id="2022">2022:</h3>
<p>What began as a surveillance system for on institution, evolved to become West Africa indispensable healthcare hub: The 4S network. Teranga is now expanding from Senegal into other West African countries: Gambia, Cape-Verde, Mauritania, Niger, Mali, Togo, Guinea, Guinea Bissau, Sierra-Leone and Benin. The goal being to provide broader coverage and improve health systems in a sustainable and effective manner.</p>
<h3 id="2023">2023:</h3>
<p>I am hired by the Institution. I'm tasked with reimplementing the current system, rebuilding it from scratch while improving on what came before. I don't have much say in the tech stack, just enough <em>freedom</em> to go with what works for <a href="https://nskm.xyz/toolbelt/">me</a>. The usual lineup of <a href="https://www.debian.org/">Debian</a>, <a href="https://www.python.org/">Python</a>, <a href="https://djangoproject.com/">Django</a>, <a href="https://www.postgresql.org/">Postgresql</a>, <a href="https://www.nginx.org/">Nginx</a>, <a href="https://podman.io">Podman</a>, <a href="https://git-scm.com/">Git</a>, <a href="https://gitlab.com/">Gitlab</a>, <a href="https://www.gnu.org/software/emacs/">Emacs</a>, <a href="https://tmuxcheatsheet.com/">Tmux</a>, etc ... More on that in the following episodes.</p>
<h3 id="2025">2025:</h3>
<p>The surveillance platform my colleagues and I have spent two years developing has now been presented to the public. But public recognition <a href="https://c.tenor.com/vLK4Mq3jiKIAAAAC/tenor.gif">is not really the source of my pride</a>. What matters is that our system, our algorithms, our data structures, our functions, our for loops, our SQL queries are actively making a difference in saving lives. Phew!</p>
<br>
<img style="display: block; margin: 0 auto; width: 250px" src="/images/wip.jpg" alt="Work in progress" title="Work in progress..."/>
<br>
<br>
<h2 id="what"><u>4S overview:</u></h2>
<blockquote>
<p>identify unusual health events earlier and provide a quicker response.</p>
</blockquote>
<p>I can't say it better.</p>
<h3 id="core">Core principles:</h3>
<p>It's really fascinating that these early systems already recognized the core principles we are trying to implement today:</p>
<h4 id="collect">1. Standardized data collection:</h4>
<p>Healthcare providers at sentinel sites should use standardized digital forms to capture essential demographic, clinical, and temporal information. We want to ensure consistency across all participating facilities and enable accurate epidemiological analysis. We should always try to adhere to WHO surveillance standards with clearly predefined entities & attributes.</p>
<h4 id="process">2. Data processing and storage:</h4>
<p>The platform should handles real-time data streaming between microservices, enabling immediate processing of incoming surveillance data. Each microservice should ensures data integrity through automated validation rules. We should always check for completeness, consistency, and quality. Validated data should be stored in a database with proper indexing for efficient retrieval.</p>
<h4 id="share">3. Information sharing:</h4>
<p>The epidemiology unit generates documents consolidating validated surveillance data for the Ministry of Health. The platform integrates with third parties to share aggregated data with the Ministry of Health. The platform also provides interactive dashboards for stakeholders.</p>
<h4 id="alert">4. Alerting and notifications:</h4>
<p>The platform should ensure immediate notification delivery to relevant actor via email and dashboard notifications. Whenever needed, the platform should automatically generates notifications through multiple channels—email alerts, in-app notifications, and dashboard warnings.</p>
<h4 id="integration">5. Integration with existing systems:</h4>
<p>The platform should seamlessly integrate with the regional health data aggregation system and any other existing platform used by Ministry of Health. The system must also integrate easily with any biobank systems that may exist.
<br></p>
<p>A drawing is worth more than thousand words:</p>
<pre data-lang="text" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-text "><code class="language-text" data-lang="text"><span> [Health facility] [Hospital] [Clinic]
</span><span> | | |
</span><span> | | |
</span><span> +----+-----------------+------------------+----+
</span><span> | 4S platform |
</span><span> | • Data collection (Patients, symptoms, ...) |
</span><span> | • Data validation & storage |
</span><span> | • Data treatment (Analyses, studies, ...) |
</span><span> | • Data sharing (Reports, alerts, ...) |
</span><span> | • Data dashboards (Statistics, plots, ...) |
</span><span> +----+-------------------+-------------+-------+
</span><span> | | |
</span><span> +----+----+ | +----+----+
</span><span> | Reports | | | Alerts |
</span><span> |Analytics| | |Warnings |
</span><span> +----+----+ | +----+----+
</span><span> | | |
</span><span> | | |
</span><span> +-----+---------+ +-----+----+ +---+-------+
</span><span> | Public Health | | Research | | Emergency |
</span><span> | Department | | | | Response |
</span><span> +---------------+ +----------+ +-----------+
</span></code></pre>
<p>This is the platform we are trying to build :) I can stop the post here, thanks for reading ...</p>
<h3 id="stack">The technological stack:</h3>
<h4 id="front">Frontend:</h4>
<p>The 4S platform frontend leverages <a href="https://angularjs.dev/">AngularJS</a> with <a href="https://www.typescriptlang.org/">TypeScript</a>, providing a dynamic and type-safe user interface that ensures smooth interactions for healthcare providers.</p>
<p>One of the main component of our architecture is <a href="https://www.keycloak.org/">Keycloak</a>, an identity & access management solution that handles user authentication, authorization accross the entire platform. Customizing the Django Rest Framework's authentication classes to handle different tokens sent by different realms was an adventure worthy of an article.</p>
<h4 id="back">Backend:</h4>
<p>The backend infrastructure centers around <a href="https://python.org">Python</a>, <a href="https://djangoproject.com">Django</a>, and Django Rest Framework. With each passing day, as we make progress on the project, I become more and more convinced that it was the right choice. Django is an amazing piece of software, thanks a lot to all its contributors.</p>
<p>Data management is handled by <a href="https://www.postgresql.org">PostgreSQL</a>, our primary database system. I can proudly say that I'm the one who made this choice. Postgresql is known for its reliability, strong adherence to SQL standards, powerful extensibility, support for complex data types, ACID compliance, robust security features, and excellent community support.</p>
<p>To handle real-time communication and event-driven processes such as email notifications, lab result updates, and system alerts, we've integrated <a href="https://kafka.apache.org/">Apache Kafka</a>. Kafka ensures that critical healthcare information flows efficiently between different system components without data loss.</p>
<p>In order to detect outbreaks, samples (blood, urine, stool, etc.) must be collected. And to effectively manage vast biological sample collections, ensure data integrity, and streamline complex workflows, we need a Biobank and a Biobanking software. A biobank is essentially an organized library of biological samples (like blood and tissue) that are stored alongside health and personal information, all used to support medical research. <a href="https://www.modul-bio.com/en/our-solutions/mbiolims/">Mbiolims</a> is the biobanking software we use to manage our samples and their associated data.</p>
<p>Here too, I would need to write a really long article telling you how we've managed to do the integration between Django and MBioLIMS :) Hehe!</p>
<h4 id="ops">Operations:</h4>
<p>All code versioning and collaboration is managed through <a href="https://git-scm.com">Git</a> & Bitbucket, ensuring secure source code management, proper access controls and audit trails. While Bitbucket provides comprehensive version control and collaboration features, I'm still convinced it was not the right choice. <a href="https://about.gitlab.com/">Gitlab</a> ? <a href="https://about.gitea.com/">Gitea</a> ? Github ? Come on!</p>
<p>The 4S platform is in fact a constallation of multiple running containers. Those containers are built using Docker and <a href="https://podman.io">Podman</a> technologies, enabling consistent deployment across different environments while simplifying development and testing workflows. I mean, I'm the only one using Podman while all my teammates are still using Docker :\</p>
<p>Our continuous integration and deployment pipeline is powered by Jenkins. Jenkins automates testing and deployment processes to maintain code quality and enable rapid, reliable updates. I prefer <a href="https://docs.gitlab.com/ci/">GitlabCI</a> by 13 parsecs.</p>
<p><a href="https://kubernetes.io/">Kubernetes</a> is used for container orchestration, deployment and operations. K8s provides automatic scaling, load balancing, and high availability to ensure our platform can handle varying loads and maintain uptime. My opinions on this tool are mixed.</p>
<p>In one side, K8s will allow us to easily scale our applications up or down based on demand. This is a critical requirement for managing sensitive healthcare information. In the other side, I live in a part of the world where most medical facilities (most people, let's be honest) do not even have a working domain name, website, and email address. How can we ask them for the DevOps expertise to manage a Kubernetes cluster? I will see to what extent I can write an article about Kubernetes.</p>
<h4 id="else">Even more:</h4>
<p>There are other essential components of the system which I am not really familiar with.</p>
<p><a href="https://dhis2.org/">DHIS2</a> is widely adopted across West Africa for integrated health information systems, disease surveillance, education management, and public health emergency response. We simply cannot ask a country that already uses DHIS2 to use the 4S platform without it being able to transfer data back to a DHIS2 instance.</p>
<p>We have an <a href="https://superset.apache.org/">Apache Superset</a> instance for data visualization and exploration. With Apache Superset, we can easily let a third party consult some information, without modifying our system.</p>
<p>Our platform generate lots of files, analysis reports & statistics. We needed something to store vast amounts of unstructured data, like images, videos, pdf documents, etc... We choosed <a href="https://github.com/minio/minio">MinIO</a> for this use case.</p>
<br>
<br>
<p>The diagram we've seen above can also be seen using this different perspective:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span> +-----------------------------------------------------------------------------------------------------------------------+
</span><span> | KUBERNETES CLUSTER |
</span><span> +-----------------------------------------------------------------------------------------------------------------------+
</span><span> | |
</span><span> | +--------------------------------------------------+ <-- (Auth/Roles) --> +------------------+ |
</span><span> | | FRONTEND CLIENTS |------------------------>| KEYCLOAK | |
</span><span> | | (AngularJS + TypeScript) |<------------------------| (Auth Server | |
</span><span> | +------------------^-------------------------------+ +-----^------------+ |
</span><span> | | | |
</span><span> | (API Requests) | (Auth/Roles) |
</span><span> | | +----+ |
</span><span> | | | |
</span><span> +---------------------v----------------------------------------------------------v--------------------------------------+
</span><span> | +--------------------------------------------------------------------------------+ +--------------------------+ |
</span><span> | | BACKEND MICROSERVICES (Django Rest Framework) | | Biobank (Mbiolims) | |
</span><span> | | Microservices exchanges data via a message broker (Kafka in our case) |<---->| Stores Samples infos | |
</span><span> | | [Microservice A] <--Data--> [Microservice B] <--Data--> [Microservice N-1] | | | |
</span><span> | +-------------------------^-----------------------^-------------------------^----+ +--------------------------+ |
</span><span> | | | | |
</span><span> | (Store/Retrieve) (Write/Open Files) (Sends data aggregations) |
</span><span> | | | | |
</span><span> +----------------------------v-----------------------v-------------------------v----------------------------------------+
</span><span> | +--------------------------------+ +--------------------------------+ +--------------------------------------------+ |
</span><span> | | POSTGRESQL DB | | MINIO | | DHIS2 | |
</span><span> | | (Primary Data Store) | | (Object/File Storage) | | (External Data Recipient) | |
</span><span> | +--------------------------------+ +--------------------------------+ +--------------------------------------------+ |
</span><span> | | (Reads Data) |
</span><span> | | |
</span><span> | +-------------v--------------------+ |
</span><span> | | APACHE SUPERSET (Dataviz) | |
</span><span> | +----------------------------------+ |
</span><span> | |
</span><span> +-----------------------------------------------------------------------------------------------------------------------+
</span></code></pre>
<br>
<br>
<h3 id="engineers">The engineering team:</h3>
<p>This article is also for my fellow developers who built the 4S network under impossible constraints—limited budgets, legacy infrastructure, unrealistic timelines. My colleagues engineers writing scripts to detect disease patterns, configuring servers that never get enough resources. My teammates who's written documentation that nobody reads but everyone depends on. I'm talking about the people who made the story:</p>
<h4 id="cisse">Cisse:</h4>
<p><a href="https://www.linkedin.com/in/mamadou-cisse-723b29114/">Cisse</a> a.k.a "Celui qui fait bouger le monde", plays a crucial role in coordinating activities across the Institution Digital Factory team. His expertise extends to managing and preserving the integrity of biomedical data within the Institution. He works closely with stakeholders to validate specifications of new health applications. To uncover more about him, follow this <a href="https://www.linkedin.com/in/mamadou-cisse-723b29114/">link</a>.</p>
<h4 id="mame-astou">Mame Astou:</h4>
<p>As an integral member of the development team, <a href="https://www.linkedin.com/in/mameastougassama">Mame Astou</a> a.k.a "Madame La Directrice", craft robust and dynamic web applications while maintaining a keen eye for intuitive user interface design. She knows how to create user experiences that prioritize both aesthetic appeal and seamless usability. What would we be without her? Absolutely nothing. To uncover more about her, please visit this <a href="https://www.linkedin.com/in/mameastougassama">page</a>.</p>
<h4 id="diom">Diom:</h4>
<p><a href="https://www.doyoubuzz.com/diom-sow">Diom</a>, a.k.a "Le Grand Marabout de l'Architecture", is a DevOps engineer who bridges the gap between development and operations. With expertise in containerization technologies, cloud platforms, and monitoring solutions, he design and maintain scalable deployment environments while ensuring system reliability and security.</p>
<h4 id="abdoul-karim">Abdoul Karim:</h4>
<p><a href="https://www.linkedin.com/in/mohamed-abdoul-karim-diop-76250b127/">Abdoul Karim</a> is probably the pillar of the team, always ready to make concessions and take responsibility. His technical experience spans multiple domains. It would take me hours to explain why he is an excellent engineer. Just go and find out by yourself how great <a href="https://www.linkedin.com/in/mohamed-abdoul-karim-diop-76250b127/">he is</a>.</p>
<h4 id="mariama">Mariama:</h4>
<p><a href="#">Mariama</a> a.k.a "La Sauveuse De Tous Les Hommes", has a deep knowledge of Scrum and agile principles. She guide the team through iterative development cycles while fostering continuous improvement and team collaboration. We are still waiting for that one day she will finally write a few lines of TypeScript. Access Mariama's comprehensive profile by following this <a href="#">link</a>.</p>
<h4 id="birante">Birante:</h4>
<p><a href="https://fr.linkedin.com/in/birantesy">Birante</a> has an exceptional expertise that encompasses everything from custom UI animations and state management to integrating native device features and third-party services. He is able to deliver native-quality experiences across both iOS, Android and Web platforms. He secretly dreams about rewriting all the 4s platform using <a href="https://www.youtube.com/watch?v=gcwzWzC7gUA">RoR</a> applications. Here is a <a href="#">link</a> to get the complete picture about Birante.</p>
<h4 id="tidiane">Tidiane:</h4>
<p><a href="https://github.com/adtidiane">Tidiane</a>, a.k.a "Le Sage", a.k.a "The Biobank expert", is probably the wisest and most reserved member of the team. Tidiane is a seasoned backend developer who brings deep expertise in Django framework development. He excel at designing robust APIs and implementing complex business logic that powers modern web applications. If you know anyone who works at <a href="https://www.modul-bio.com">Mbiolims</a>, please tell them they should hire Tidiane. His background can be found <a href="https://www.linkedin.com/in/amadou-tidiane-diallo-61065a41/">here</a>.</p>
<h4 id="adrien">Adrien:</h4>
<p>Aux âmes bien nées, la valeur n'attend point le nombre d'années. <a href="#">Adrien</a>, the youngest in the team, is a software engineer with an extensive experience in developing RESTful APIs, managing database migrations, and implementing automated testing frameworks within Django projects. For additional information about Adrien, visit this <a href="#">link</a>.</p>
<p><strong>Guys, your dedication, your resilience, your ingenuity deserve recognition far beyond what they receive.
Thank you.</strong>
<br>
<br></p>
<h3 id="future">The roadmap:</h3>
<p>Right now we're developing comprehensive features related to <strong>"case management"</strong>. Those features will enable healthcare providers to track patient journeys from initial consultation through treatment completion, ensuring continuity of care and improved outcomes.</p>
<p>We plan to extend our platform to handle a <strong>wider spectrum of medical conditions and syndromes</strong>, transforming it from a specialized tool into a comprehensive healthcare management solution.</p>
<p>Our platform will also align with the <strong>One Health approach</strong>, recognizing the interconnection between human, animal, and environmental health.</p>
<p>Data exchange with other healthcare systems, electronic health records, and medical devices, is something we need to work towards. We really need to meet evolving healthcare standards and prepare for international expansion. While I agree we didn't put enough efforts on this, compliance and interoperability remain central to our development strategy.</p>
<h2 id="more"><u>More on the topic:</u></h2>
<p>Writing about 4S implementation could fill volumes. For those ready to dig deeper into the technical and policy dimensions of public health surveillance, <em>and before I write the second part of this series</em>, here are resources that expand on the key concepts discussed:</p>
<ul>
<li><a href="https://medium.com/healthsites-io/cartographie-des-donn%C3%A9es-sanitaires-durgence-au-s%C3%A9n%C3%A9gal-474c3c3bec29">Mapping emergency health data in Senegal</a></li>
<li><a href="https://www.youtube.com/watch?v=Z8pa6BQN54w">How CARE transforms outbreaks into opportunities for Africa</a></li>
<li><a href="https://www.youtube.com/watch?v=cZfzP3J2VxY">A new way to fight infectious diseases and predict outbreaks</a></li>
<li><a href="https://stories.theglobalfund.org/early-detection-for-rapid-response-strengthening-disease-surveillance-across-west-africa">Early detection for rapid response: strengthening disease surveillance across West Africa</a></li>
<li><a href="https://institutpasteurdakar.sn/strengthening-epidemic-resilience-through-one-health-and-integrated-community-based-surveillance/">Strengthening epidemic resilience through one health and integrated community-based surveillance</a></li>
<li><a href="https://www.kaikai.dev/blog/blog-1/ihe-for-senegal-a-look-back-at-the-ihe-training-week-6">Launch and training on medical data exchange in Senegal according to IHE standards</a></li>
<li><a href="https://www.youtube.com/watch?v=pmiQza2kQmE">Dengue sylvatique : l’angle mort du diagnostic — Dr Idrissa Dieng</a></li>
</ul>
<p>Some people we might want to collaborate with:</p>
<ul>
<li><a href="https://data.humdata.org/">the Humanitarian Data Exchange</a></li>
<li><a href="https://healthsites.io/">HealthSites.IO, I've contributed Python code for them</a></li>
<li><a href="https://eyone.net/en/about-us">Eyone, improving the quality of medical care through technology.</a></li>
<li><a href="https://njureel.com/#accueil">Njureel, reinventing the proximity between healthcare professionals and patients</a></li>
<li><a href="https://www.geomatica.space/">Geomatica, one of the greatest company in geospatial data (The founder is a very good friend)</a></li>
</ul>
<p>I hope you learned something. If you found this article interesting, please share it <a href="https://www.youtube.com/watch?v=K0HSD_i2DvA">around the world</a>. Tell them there are people here who save lives with free and open source software.</p>
<br>
<img style="display: block; margin: 0 auto; width: 250px" src="/images/sample.jpg" alt="" title="Sample container"/>
<br>
<br>
Free and Open Source Software.... Again!2025-05-31T00:00:00+00:002025-05-31T00:00:00+00:00
Unknown
https://nskm.xyz/posts/fossa/ <img style="display: block; margin: 0 auto; width: 400px" src="/images/liberty.jpg" alt="" title="Liberty"/>
<br>
<br>
<h1 id="table-of-content">Table Of Content:</h1>
<ul>
<li><a href="https://nskm.xyz/posts/fossa/#1">1 Intro </a></li>
<li><a href="https://nskm.xyz/posts/fossa/#4">2. Why contribute</a></li>
<li><a href="https://nskm.xyz/posts/fossa/#13">3. Skepticism</a></li>
<li><a href="https://nskm.xyz/posts/fossa/#14">4. How to contribute</a></li>
<li><a href="https://nskm.xyz/posts/fossa/#15">5. Success stories</a></li>
<li><a href="https://nskm.xyz/posts/fossa/#16">6. More on the topic</a></li>
</ul>
<h2 id="1">Intro</h2>
<p>I'm tired of seeing that the majority of Africans still don't <a href="#">get it</a>, and probably never will, that free and open source software represents the optimal path forward for our continent. <a href="https://roylab.xyz/">One of</a> my brightest students asked me to talk about free and open source software, and gave me 2 hours to do so. For him, for Africa, for the future, <a href="https://c.tenor.com/h2M4FDyKANQAAAAC/tenor.gif">here I am</a>...</p>
<p>IT's been almost 20 years I see myself as a <a href="https://lwn.net/SubscriberLink/1023299/7ba649b1ede41895/">free software</a> advocate, therefore my perspectives are inherently biased. Hopefully, the bias is not huge. Let's cue the <a href="https://www.youtube.com/watch?v=UAs9-VL66vo">music</a> please!</p>
<h3 id="2"><ins><em>Free software:</em></ins></h3>
<p>The Free software movement, as defined by the <a href="https://www.fsf.org/">Free Software Foundation</a>, is a social/political movement about freedom, ethics, and user rights/freedoms. Richard Stallman created the movement in 1983 and the <a href="https://www.gnu.org/philosophy/free-sw.html#four-freedoms">4 essential user freedoms are</a>:</p>
<ul>
<li>Freedom to run the program for any purpose</li>
<li>Freedom to study how the program works and change it</li>
<li>Freedom to redistribute copies to help others</li>
<li>Freedom to distribute modified versions to benefit the community</li>
</ul>
<h3 id="3"><ins><em>Open source software:</em></ins></h3>
<p>The term "open source" was coined to rebrand free software with a business-friendly focus. It emphasizes practical benefits like better quality, reliability, and development methodology rather than ethical considerations. Bruce Perens created the <a href="https://opensource.org/">Open Source Initiative</a> in 1998 and Open source criteria include:</p>
<ul>
<li>Source code must be available</li>
<li>Free redistribution</li>
<li>Derived works allowed</li>
<li>No discrimination against groups or fields of use</li>
<li>License must not restrict other software</li>
</ul>
<p>If we read these <em>two definitions</em> calmly, we can see that there is a very important nuance that should be made here: <strong>the user freedom and independence</strong> is at the center of everything for a free software advocate. This is not necessarily the case for an open source advocate.</p>
<br>
<h2 id="4">Why contribute:</h2>
<p>Here are compelling reasons why African people should fully embrace Free and Open Source software :</p>
<h3 id="5"><ins><em>Learning:</em></ins></h3>
<p>Free and Open Source Software often attract talented developers. When your code get reviewed by maintainers at companies around the globe, you receive mentorship that money can't buy. You'll encounter diverse codebases, architectural patterns, problem-solving approaches that you might never see in your day job. Free and Open Source Software forces you to read and understand existing code, which dramatically improves your development skills.</p>
<h3 id="6"><ins><em>Economic opportunities:</em></ins></h3>
<p>Open source contributions can help you land jobs or consulting opportunities. There are many companies worldwide are actively seeking talents. Open source provides the perfect <a href="https://www.wearedevelopers.com/world-congress/github-open-source-spotlight">showcase</a> of your <a href="https://events.linuxfoundation.org/open-source-summit-europe/">abilities</a>. Open source software is often the most accessible option for people with limited budgets. There are a lot of startups across the Black Continent that rely on free and open source solutions to build their digital infrastructure.</p>
<p>We often talk about <a href="https://www.redhat.com/en/about/development-model">RedHat</a>, but there are many other companies like: <a href="https://www.zdnet.com/article/open-source-is-getting-bigger-and-richer-says-suse/">Suse</a>, <a href="https://about.gitlab.com/blog/monetizing-and-being-open-source/">Gitlab</a>, <a href="https://news.ycombinator.com/item?id=13774420">Docker</a>, <a href="https://nextcloud.com/about/">NextCloud</a>, companies that demonstrate various and successful approaches built on free and open source foundations.</p>
<p>In a nutshell, those companies will offer subscriptions and services for a guaranteed support, security updates, patches, custom implementations, customizations, optimizations, trainings, certifications, consulting, ...</p>
<p>I would like to do a special mention for 2 companies: <a href="https://www.entrouvert.com/expertise/logiciels-reellement-libres/">Entr'Ouvert</a> and <a href="https://www.logilab.fr/societe">Logilab</a>. They offer implementations, customizations, trainings, and consulting services exclusively with free software, you can see it <a href="https://dev.entrouvert.org/">here</a> and <a href="https://forge.extranet.logilab.fr/explore">there</a>. They have a deep expertise in the free software industry and this expertise became a valuable service to their clients. Also, I have very good friends over there, shoutout o/</p>
<h3 id="7"><ins><em>Breaking geographic barriers:</em></ins></h3>
<p>Thanks to Free and Open Source Software, your code quality matters more than your location or background. African developers can showcase their skills on the same <a href="https://github.com">platforms</a> as Silicon Valley engineers. Free and Open Source is our opportunity for challenging stereotypes and demonstrating our technical talent to the world.</p>
<h3 id="78"><ins><em>Gaming:</em></ins></h3>
<p>Linux was once held back by its compatibility with PC games, but thanks to <a href="https://www.valvesoftware.com">Valve</a> and <a href="https://store.steampowered.com/steamos/">SteamOS</a>, those days are in the past. Better gaming performance, better battery life, better power management, are <a href="https://www.youtube.com/watch?v=CJXp3UYj50Q">another reasons</a> to use Linux.</p>
<h3 id="8"><ins><em>Community:</em></ins></h3>
<p>The open source community represents humanity at its best, a global network where people freely exchange expertise and genuinely celebrate one another's wins. When you realize your contribution has solved someone's problem or saved them hours of work, that feeling is pure gold. Belonging to a community, building <a href="https://www.thesocialcreatures.org/thecreaturetimes/evolution-of-social-connection">social connections</a> is essential to our wellbeing, it is as important as food, water and shelter.</p>
<h3 id="9"><ins><em>Digital sovereignty:</em></ins></h3>
<p><a href="https://arxiv.org/abs/2307.01791">Digital sovereignty</a> encompasses control over critical digital infrastructure like internet networks, data centers, communication systems and mobile/web platforms. Digital sovereignty is the ability to regulate how <em>our</em> data is collected, stored, and processed within <em>our</em> jurisdiction.</p>
<p>What happens when a country's government systems, banking infrastructure, or communication networks depend entirely on software and services controlled by foreign companies. This is where free and open source software becomes crucial for digital sovereignty.</p>
<p>When you use non free and non open source software:</p>
<ul>
<li>you're dependent on the vendor's decisions, pricing, and continued support.</li>
<li>you can't see how it works, modify it for your needs, or ensure it doesn't contain <a href="https://cyberlaw.ccdcoe.org/wiki/African_Union_headquarters_hack_(2018)">backdoors</a></li>
<li>you <a href="https://www.dw.com/en/will-meta-lawsuits-shape-africas-data-privacy-laws/a-72567105">lose the right to determine how your data is used</a></li>
</ul>
<h3 id="10"><ins><em>The others are moving</em>:</ins></h3>
<p><a href="https://publiccode.eu/en/">Public money, public code</a> an initiative advocating that software developed with public funds should be made freely available under a free and open-source license.</p>
<p><a href="https://code.gouv.fr/fr/">Pôle open source et communs numériques</a>, Initiative to support government administrations in using free and open source software, assist their efforts to publish and share source code, and create connections with the open source ecosystem.</p>
<p><a href="https://blog.documentfoundation.org/blog/2024/04/04/german-state-moving-30000-pcs-to-libreoffice/">The German federal state</a> has decided to move from Microsoft Windows and Microsoft Office to Linux and LibreOffice (and other free and open source software) on the 30,000 PCs used in the local government.</p>
<p><a href="https://eu-os.eu/">EU OS</a>, a proof-of-concept for an Operating System for a typical organization inside the European Union public sector.</p>
<p>To move away from dependence on US software solutions and acquire genuine digital sovereignty, the <a href="https://www.lyon.fr/actualite/action-municipale/la-ville-de-lyon-renforce-sa-souverainete-numerique">City of Lyon</a> has embarked on a major transformation of its digital tools.</p>
<p><em>At this point, I don't even know what else should I say to convince you...</em></p>
<p><a href="https://www.zdnet.com/article/why-denmark-is-dumping-microsoft-office-and-windows-for-libreoffice-and-linux/">Denmark</a> is transitioning away from Microsoft products in a key ministry due to a strategy focused on digital sovereignty and reducing reliance on foreign, particularly US-based, tech giants. This involves replacing Windows and Office 365 with open-source alternatives like Linux and LibreOffice.</p>
<p>The <a href="https://economictimes.indiatimes.com/news/international/us/after-danish-cities-germanys-schleswig-holstein-state-government-to-ban-microsoft-programs-at-work/articleshow/121833653.cms">Schleswig-Holstein state</a> government is shifting completely to free and open source software, aiming to achieve what officials call "digital sovereignty" and to reduce reliance on U.S.-based tech giants [...] This isn’t just about software. It's about control over sensitive data.</p>
<h3 id="11"><ins><em>You're already consuming:</em></ins></h3>
<p>If you've ever used <a href="https://opensource.google/">Google</a>, <a href="https://opensource.fb.com/">Facebook</a>, <a href="https://github.com/whatsapp">WhatsApp</a>, <a href="https://opensource.microsoft.com/">Microsoft</a> or watched a <a href="https://netflix.github.io/">Netflix</a> movie today, you've been relying on free and open source software without even knowing it. These tech giants don't reinvent the wheel, they build their services on top of thousands of open source projects. None of those things would exist without free and open source software.</p>
<br>
<h2 id="13">Skepticism:</h2>
<img style="display: block; margin: 0 auto; width: 300px" src="/images/skeptic.jpg" alt="skeptical" title="If free software is so great, why do I get the feeling that no one uses it?"/>
<p>Technological proficiency is important, but <strong>correctly managing the cultural</strong> shift required to embrace free and open source software is also very important. There are a number of <a href="https://www.youtube.com/watch?v=5GXGdavLu6Y">complex reasons</a> why mass adoption of free and open software has not yet taken place in Africa.</p>
<h3 id="cultural-resistance-and-habits"><ins><em>Cultural resistance and habits:</em></ins></h3>
<p>Users are used to the interfaces and workflows of proprietary software. Change naturally generates resistance, especially when the benefits are not immediately perceptible to the end-user.</p>
<h3 id="institutional-inertia"><ins><em>Institutional inertia:</em></ins></h3>
<p>Public and private organizations are often prisoners of their past technological choices. Migrating complex systems requires massive investment in training, data migration and process adaptation. This <a href="https://www.youtube.com/shorts/7RXK-nR83f4">vendor lock-in</a> is particularly strong in public administration.</p>
<h3 id="perception-of-complexity-and-support"><ins><em>Perception of complexity and support:</em></ins></h3>
<p>In my humble opinion, this perception no longer corresponds to current reality. Yet, many still consider free and open source software to be more technical and less user-friendly.</p>
<h3 id="lobbying-and-political-influence"><ins><em>Lobbying and political influence:</em></ins></h3>
<p>Large technology companies exert considerable <a href="https://corporateeurope.org/en/2023/09/lobbying-power-amazon-google-and-co-continues-grow">influence</a> and <a href="https://www.lbc.co.uk/world-news/british-icc-chief-prosecutor-lost-email-bank-accounts-frozen-trump-sanctions/">pressure</a> on political decisions and public procurement, hindering the adoption of alternative solutions.</p>
<h3 id="economic-reasons"><ins><em>Economic reasons:</em></ins></h3>
<p>Business models built around proprietary software generate huge revenues. Companies like Microsoft, Adobe or Oracle have built entire ecosystems that create mutual financial dependency. To change would represent considerable transition costs and a loss of revenue for many players.</p>
<h3 id="to-go-even-deeper-on-this-subject"><ins><em>To go even deeper on this subject:</em></ins></h3>
<p>Here is a really interesting <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3138085">study</a>, intitled <strong><em>Adoption Barriers of Open-Source Software: A Systematic Review</em></strong> and led by <a href="https://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=2321721">Dmitrij Petrov</a> & <a href="https://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=2544892">Nikolaus Obwegeser</a>. A study that tried to find all the significant barriers that hinder the wider adoption of Free and Open source software. Have been found 19 high-level factors that challenge or inhibit the adoption of FLOSS in various contexts. Long story short....</p>
<br>
<img style="display: block; margin: 0 auto; width: 900px" src="/images/adoption_barriers_of_foss.png" alt="" title="Barriers of FOSS"/>
<br>
<br>
<h2 id="14">How to contribute:</h2>
<p>There are so many <a href="https://contribute.cncf.io/contributors/getting-started/">ways</a> to <a href="https://docs.github.com/en/get-started/exploring-projects-on-github/finding-ways-to-contribute-to-open-source-on-github">contribute</a> to free and open source software. There are so many <a href="https://www.freecodecamp.org/news/how-to-contribute-to-open-source-projects-beginners-guide/#:~:text=Adding%20a%20description%20to%20a,the%20project%27s%20work%20folder%20correctly.">articles</a>, so many <a href="https://opensource.guide/how-to-contribute/">guides</a> and <a href="https://dev.to/opensauced/how-i-got-hired-contributing-to-open-source-projects-546i">tutorials</a>. The choice is <a href="https://www.digitalocean.com/resources/articles/how-to-contribute-to-open-source">yours</a>.</p>
<p>If you really want me to answer the question, this is more or less how it should be done (on top of my head).</p>
<h4 id="0-contribution-types">0. Contribution types:</h4>
<p>There are several ways you can contribute to free and open source projects.</p>
<p>Bug fixes involve addressing opened issues, improving error handling, and resolving technical problems that users have encountered. Feature contributions include adding new functionality, enhancing user experience and interface design, and optimizing performance to make applications run more efficiently.</p>
<p><a href="https://diataxis.fr/">Documentation</a> improvements focus on enhancing readability, adding helpful examples and tutorials, providing translations for broader accessibility, and fixing typos or grammatical errors.</p>
<p>The last contribution type I would like to talk about is funding. Funding through <a href="https://my.fsf.org/join">donating</a> or direct <a href="https://palletsprojects.com/donate">sponsoring</a> can provide crucial financial support to help free and open source projects continue their development.</p>
<h4 id="1-tips">1. Tips:</h4>
<p>Before diving into contributions, it's essential to read the <a href="https://github.com/django/django/blob/main/CONTRIBUTING.rst">CONTRIBUTING.md</a> file or any other available documentation to understand the project's specific guidelines and processes.</p>
<p>Starting small with simple fixes like correcting typos or <a href="https://github.com/numpy/numpy/issues/29258">improving documentation</a> can be incredibly valuable and help you get familiar with the contribution workflow.</p>
<p>Remember to be patient during the review process, as <a href="https://github.com/django/django/pull/19565">reviews</a> are conducted by human beings who need time to thoroughly examine your work. Don't hesitate to ask <a href="https://www.youtube.com/watch?v=5k0ndcJiECo">questions</a> when you're stuck, and feel free to ask even more <a href="https://palletsprojects.com/contributing/questions">questions</a> to clarify requirements or processes.</p>
<p>It's also important to focus on making one logical change per Pull Request to keep your contributions clear and manageable for reviewers.</p>
<h4 id="2-discover-explore">2. Discover & explore:</h4>
<p>Begin your open source journey by <a href="https://github.com/firstcontributions/first-contributions">finding</a> a project that you <strong>actively use</strong> or <strong>genuinely care about</strong>, as this personal connection will help sustain your <strong>motivation</strong>. Take time to thoroughly read the project's documentation and contributing guidelines to understand how the community operates and what standards they maintain. Browse through the <a href="https://github.com/numpy/numpy/issues">project's issues</a> section, paying special attention to those tagged with <a href="https://github.com/jazzband/django-silk/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22">"good first issue"</a> or "beginner" labels, as these are specifically designed to help newcomers get started with meaningful contributions.</p>
<h4 id="3-setup-your-environment">3. Setup your environment:</h4>
<p>Getting started with development requires setting up your workspace properly. First, <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo">fork</a> the repository to create your own copy of the project. Next, <a href="https://git-scm.com/docs/git-clone">clone</a> this forked repository to your local machine where you can make changes and test your work.</p>
<p>Take time to set up the development environment according to the project's instructions, which may involve <a href="#">installing</a> specific dependencies or configuring tools. Run any available <a href="#">unit tests</a> to ensure everything is working correctly in your setup, as this confirms your environment is properly configured. Finally, <a href="https://git-scm.com/docs/git-switch">create a new branch</a> for your work to keep your changes organized and separate from the main codebase.</p>
<h4 id="4-make-your-contribution">4. Make your contribution:</h4>
<p>When working on your contribution, it's crucial to follow the project's established <a href="https://peps.python.org/pep-0008/">coding style and conventions</a> to maintain consistency across the codebase. Focus on writing your code, fixing bugs, or adding features while keeping the project's standards and best practices in mind. Remember to update <a href="#">documentation</a> whenever your changes affect how users or developers interact with the project. Additionally, add or update <a href="#">tests</a> to cover your changes, ensuring that new functionality works as expected and that bug fixes prevent regression.</p>
<h4 id="5-test-prepare">5. Test & prepare</h4>
<p>Before submitting your work, thoroughly <a href="https://github.com/django/django/tree/main/tests">test</a> your changes by running tests locally to catch any issues early in the process. Carefully review your changes to ensure they meet the project's quality standards and actually solve the intended problem.</p>
<p>Write clear, <a href="https://www.conventionalcommits.org/en/v1.0.0/">descriptive commit messages</a> that explain what your changes do and why they were necessary, as this helps maintainers understand your contribution. Double-check your work against the <a href="https://docs.djangoproject.com/en/dev/internals/contributing/">contributing guide</a> to make sure you've followed all the project's requirements and conventions.</p>
<h4 id="6-submit-your-work">6. Submit your work</h4>
<p>Once your contribution is ready, <a href="https://git-scm.com/docs/git-push">push</a> your branch to your forked repository to make it available for review. Create a <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests">Pull Request</a> from your branch to the original project's main branch, providing a clear pathway for your changes to be reviewed and potentially merged.</p>
<p>Fill out the PR template with detailed information about what your changes do, why they're needed, and how they've been tested. If your contribution addresses specific issues, link those related issues in your PR description to provide context and help maintainers track the resolution of reported problems.</p>
<h4 id="7-collaborate-refine">7. Collaborate & refine</h4>
<p>After submitting your Pull Request, <a href="https://github.com/healthsites/healthsites/pulls?q=+is%3Apr+author%3Alemeteore+">actively engage with the review process</a> by responding promptly and thoughtfully to code review feedback from maintainers and other contributors. Make any requested changes to address concerns or suggestions, treating each piece of feedback as an opportunity to improve your contribution and learn from experienced developers.</p>
<p>Throughout this process, engage in discussions <strong>respectfully and professionally</strong>, remembering that constructive feedback is meant to help improve the project rather than criticize your work personally. Pay attention to any failing CI checks and address them promptly, as these automated tests help ensure your changes don't break existing functionality.</p>
<h4 id="8-merge-celebrate">8. Merge & celebrate:</h4>
<p>Your contribution gets merged into the main project, congratulations 🎉. This marks a significant achievement in your open source journey. You're now officially a project contributor, joining a community of developers who have helped shape and improve the software. Most importantly, your code now helps users worldwide, potentially solving problems, adding useful features, or improving the experience for countless people who rely on the project.</p>
<p>Here is an <a href="https://www.cncf.io/blog/2023/04/03/outlining-the-structure-of-your-open-source-software-project/">article</a> discussing, among other things, the tasks that are needed to be taken care of for different sizes of the open source project. We see that the bigger the project, the more ways there are to contribute.</p>
<h2 id="15">Success stories:</h2>
<p>After <a href="https://x.com/fossfa">searching</a> <a href="https://www.crunchbase.com/organization/fossfa">very</a> <a href="https://github.com/FOSSFA">extensively</a> <a href="https://www.instagram.com/fossfa/">everywhere</a> on the internet, I found no <strong>current</strong> organizations in Africa, that even remotely resembles the Free Software Foundation.</p>
<p>If you find interesting (and recent) informations about free and open source software initiaves/organizations in Africa, if you are in one way or another, linked to an active Linux User Group in Africa, please, if you know any African initiative/organization that promotes free and open source software in Africa, please let me know. I'll be glad to update the article.</p>
<br>
<img style="display: block; margin: 0 auto; width: 500px" src="/images/open-source-contributions.jpg" alt="" title="Hopefully, you won't leave! Because we need your help!"/>
<h2 id="16">More on the topic:</h2>
<p>We could easily spend days on this topic! Since we only have two hours, here are some excellent resources to continue your journey into this fascinating subject.</p>
<p>In french:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=Ef8jkpJEDVc">La liberté du logiciel expliquée en moins de 3 min!</a></li>
<li><a href="https://www.youtube.com/watch?v=Xazuo7Rva2Y">Se lancer dans l'open source à 100% (et en vivre !)</a></li>
<li><a href="https://www.youtube.com/watch?v=JWJmFTcH1Bw">Le logiciel libre comme levier de souveraineté</a></li>
<li><a href="https://www.youtube.com/watch?v=YQTEq9BRefo">Open Source: Comprendre, Contribuer</a></li>
<li><a href="https://www.youtube.com/watch?v=y4AjuV0O3wo">Comment devenir un contributeur Open Source ?</a></li>
<li><a href="https://www.youtube.com/watch?v=8U80yLOZLp4">Open source & logiciel libre</a></li>
</ul>
<p>In english:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=PVD1LNDxOnc">Open Source explained</a></li>
<li><a href="https://opensource.guide/how-to-contribute/">How to contribute to Open Source</a></li>
<li><a href="https://www.digitalocean.com/resources/articles/how-to-contribute-to-open-source">How to contribute to Open Source, a step by step guide</a></li>
<li><a href="https://www.gnu.org/philosophy/open-source-misses-the-point.en.html">Why Open Source misses the point of Free Software</a></li>
<li><a href="https://lwn.net/Articles/1013776/">Lessons from open source in the Mexican government</a></li>
</ul>
<p>Let's be optimistic:</p>
<ul>
<li><a href="https://livingopensource.org/the-rise-of-open-source-software-in-africa-current-trends-and-futureprospects/">The rise of Open Source software in Africa: current trends and future prospects</a></li>
<li><a href="https://www.africantechroundup.com/open-source-proprietary-software-african-tech-round-up/">Open Source vs proprietary software: what is best for Africa?</a></li>
<li><a href="https://www.ictworks.org/digital-colonialism-african-software-development/">How to break digital colonialism in African software development</a></li>
<li><a href="https://www.universityworldnews.com/post.php?story=20241203094802197">Ten university teams worked on open-source operating system</a></li>
</ul>
<p>By the way...:</p>
<ul>
<li>The Free Software Foundation (FSF) <a href="https://www.fsf.org/fsf40">turns forty</a> this year</li>
<li>PyCon Africa and PyConZA are <a href="https://za.pycon.org/">joining forces!</a></li>
<li><a href="https://2025.djangocon.africa/">DjangoCon Africa</a></li>
<li>Co-locating <a href="https://discourse.ubuntu.com/t/ubucon-africa-djangocon-africa-2025/59940">the 1st UbuCon Africa with the passionate Django community!</a></li>
<li><a href="https://oshwa.org/about/">The Open Source Hardware Association</a></li>
<li>Framework <a href="https://frame.work/about">electronics are open to repair and upgrade</a></li>
</ul>
<p>In the next few weeks, I will publish findings regarding the <a href="https://nskm.xyz/posts/ssssss/">Syndromic Surveillance System</a> I'm currently working on. Expect a detailed writeup showing exactly how free and open source software is driving a positive change to the West Africa's public health infrastructure. Stay tuned...</p>
<br>
<img style="display: block; margin: 0 auto; width: 200px" src="/images/stay_tuned.gif" alt="" title="stay tuned"/>
<p>Thanks for sharing!</p>
C Pointers2024-06-18T00:00:00+00:002024-06-18T00:00:00+00:00
Unknown
https://nskm.xyz/posts/ptrs/<p>In my <a href="https://nskm.xyz/posts/cp/">last</a> post, I've written about lots of things but not about C pointers. I thought the topic deserves a post in itself because it's one of the most important features of the C language.</p>
<p>Let's start with some <a href="https://yewtu.be/watch?v=24ntpKNiUMs">music</a> please.</p>
<img style="display: block; margin: 0 auto; width: 550px" src="/assets/pointers.png" alt="C Pointers" title="C Pointers"/>
<br>
<br>
<h2 id="toc">ToC</h2>
<ul>
<li><a href="https://nskm.xyz/posts/ptrs/#i">Introduction</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#d">Declaration & initialization</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#o">Operators</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#u">Use cases</a>
<ul>
<li><a href="https://nskm.xyz/posts/ptrs/#a">Arrays & strings manipulation</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#p">Pass by reference</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#m">Memory allocation</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#s">Data structure implementation</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#f">Functions pointers</a></li>
<li><a href="https://nskm.xyz/posts/ptrs/#h">Hardware interface</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/ptrs/#g">Going further</a></li>
</ul>
<h2 id="i">Introduction:</h2>
<p>A pointer is a variable that contains the memory address of another variable. A pointer allows
us to <em>indirectly</em> access and manipulate the value of the variable that is pointed to. Like other variables, C pointers have a type.</p>
<p><img
style="display: block; margin: 0 auto; width: 600px"
src="/assets/pointers_repr.png"
alt="Pointers"
title="Pointers"/></p>
<center>
<a href="https://codeforwin.org">Source</a>
</center>
<h2 id="d">Declaration and initialization:</h2>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main () {
</span><span> </span><span style="color:#569cd6;">int</span><span> foo = </span><span style="color:#b5cea8;">42</span><span>;
</span><span> </span><span style="color:#569cd6;">int</span><span> bar = </span><span style="color:#b5cea8;">83</span><span>;
</span><span> </span><span style="color:#569cd6;">int *</span><span>ptr = </span><span style="color:#569cd6;">&</span><span>foo; </span><span style="color:#608b4e;">// Pointer variable, type int, receives the address of foo variable*
</span><span> </span><span style="color:#569cd6;">int **</span><span>ptrptr = </span><span style="color:#569cd6;">&</span><span>ptr; </span><span style="color:#608b4e;">// pointer to a pointer
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Value of foo: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, foo);
</span><span> printf(</span><span style="color:#d69d85;">"Value of ptr is: </span><span style="color:#b4cea8;">%p</span><span style="color:#d69d85;">, and ptr is pointing to: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">void *</span><span>)ptr, </span><span style="color:#569cd6;">*</span><span>ptr);
</span><span> printf(</span><span style="color:#d69d85;">"Value of ptrptr is: </span><span style="color:#b4cea8;">%p</span><span style="color:#d69d85;"> & ptrptr is pointing to: </span><span style="color:#b4cea8;">%p</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">void *</span><span>)ptrptr, (</span><span style="color:#569cd6;">void *</span><span>)ptr);
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> </span><span style="color:#569cd6;">*</span><span>ptr = </span><span style="color:#b5cea8;">12</span><span>;
</span><span> printf(</span><span style="color:#d69d85;">"Value of foo after modification via ptr: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, foo);
</span><span> </span><span style="color:#569cd6;">**</span><span>ptrptr = </span><span style="color:#b5cea8;">13</span><span>;
</span><span> printf(</span><span style="color:#d69d85;">"Value of foo after modification via ptr to ptr: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, foo);
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> ptr = </span><span style="color:#569cd6;">&</span><span>bar;
</span><span> printf(</span><span style="color:#d69d85;">"Value of ptr is now: </span><span style="color:#b4cea8;">%p</span><span style="color:#d69d85;">, and ptr is now pointing to: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">void *</span><span>)ptr, </span><span style="color:#569cd6;">*</span><span>ptr);
</span><span> printf(</span><span style="color:#d69d85;">"Value of ptrptr, still: </span><span style="color:#b4cea8;">%p</span><span style="color:#d69d85;"> & ptrptr is now pointing to: </span><span style="color:#b4cea8;">%p</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">void *</span><span>)ptrptr, (</span><span style="color:#569cd6;">void *</span><span>)ptr);
</span><span>
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<p>And the output of the previous program is:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>Value of foo: 42
</span><span>Value of ptr is: 0x7ffc6155fd60, and ptr is pointing to: 42
</span><span>Value of ptrptr is: 0x7ffc6155fd68 </span><span style="color:#569cd6;">& </span><span>ptrptr is pointing to: 0x7ffc6155fd60
</span><span>
</span><span>Value of foo after modification via ptr: 12
</span><span>Value of foo after modification via ptr to ptr: 13
</span><span>
</span><span>Value of ptr is now: 0x7ffc6155fd64, and ptr is now pointing to: 83
</span><span>Value of ptrptr, still: 0x7ffc6155fd68 </span><span style="color:#569cd6;">& </span><span>ptrptr is now pointing to: 0x7ffc6155fd64
</span></code></pre>
<h2 id="o">C pointer operators:</h2>
<p>C offers operators for pointer manipulation:</p>
<ul>
<li>Address-of operator (&): to return the memory address of a variable</li>
<li>Dereference operator (*): to access the value stored at the memory address pointed to</li>
<li>Arithmetic operators: to perform arithmetic operations on C pointers, addition (+), subtraction (-), increment (++), and decrement (--)</li>
</ul>
<h2 id="u">Use cases for C pointers:</h2>
<p>All modern programming languages have pointers in some form or other. Pointers are used everywhere there is a program, even when we don't see them. Let's explore some of those use cases.</p>
<h3 id="a">Manipulating array and string elements:</h3>
<p>C Pointers can be used to access and manipulate elements in arrays and strings. C Pointers provide a way to efficiently iterate over arrays and perform operations on each item of the sequence.</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#569cd6;">int</span><span> arr[</span><span style="color:#b5cea8;">5</span><span>] = {</span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#b5cea8;">2</span><span>, </span><span style="color:#b5cea8;">3</span><span>, </span><span style="color:#b5cea8;">4</span><span>, </span><span style="color:#b5cea8;">5</span><span>};
</span><span style="color:#569cd6;">int *</span><span>aptr = arr;
</span><span>
</span><span style="color:#608b4e;">// print items of the sequence using array indices
</span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> i = </span><span style="color:#b5cea8;">0</span><span>; i < </span><span style="color:#b5cea8;">5</span><span>; i++) {
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d </span><span style="color:#d69d85;">"</span><span>, arr[i]);
</span><span>}
</span><span>printf(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span>
</span><span style="color:#608b4e;">// print elements of the array using pointer
</span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> i = </span><span style="color:#b5cea8;">0</span><span>; i < </span><span style="color:#b5cea8;">5</span><span>; i++) {
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d </span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>aptr);
</span><span> aptr++;
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// print items of the sequence using pointer arithmetic
</span><span style="color:#608b4e;">// and the fact that arr is actually a pointer to the first item of the arr
</span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> i = </span><span style="color:#b5cea8;">0</span><span>; i < </span><span style="color:#b5cea8;">5</span><span>; i++) {
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d </span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>arr+i);
</span><span>}
</span></code></pre>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#608b4e;">// an array of characters
</span><span style="color:#569cd6;">char</span><span> str[] = </span><span style="color:#d69d85;">"Hello world from West Africa"</span><span>;
</span><span style="color:#569cd6;">char *</span><span>sptr = str; </span><span style="color:#608b4e;">// a pointer to the first element of the array
</span><span>
</span><span style="color:#569cd6;">int</span><span> i = </span><span style="color:#b5cea8;">0</span><span>;
</span><span style="color:#569cd6;">while</span><span>(i < </span><span style="color:#b5cea8;">28</span><span>){
</span><span> </span><span style="color:#608b4e;">// while i is less than 28, print the character at i-th index
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%c </span><span style="color:#d69d85;">"</span><span>, str[i]);
</span><span> </span><span style="color:#608b4e;">// increment i to go to the next index
</span><span> i++;
</span><span> }
</span><span>
</span><span>printf(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span>
</span><span style="color:#608b4e;">// while the value that is pointed to is different from the "\0" character
</span><span style="color:#569cd6;">while </span><span>(</span><span style="color:#569cd6;">*</span><span>sptr != </span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\0</span><span style="color:#d69d85;">'</span><span>) {
</span><span> </span><span style="color:#608b4e;">// print the char
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%c </span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>sptr);
</span><span> </span><span style="color:#608b4e;">// increment pointer (to go to the next character)
</span><span> sptr = sptr + </span><span style="color:#b5cea8;">1</span><span>;
</span><span>}
</span><span>
</span><span>printf(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span>
</span><span style="color:#569cd6;">int</span><span> m = </span><span style="color:#b5cea8;">0</span><span>;
</span><span style="color:#569cd6;">while</span><span>(m < </span><span style="color:#b5cea8;">28</span><span>){
</span><span> </span><span style="color:#608b4e;">// str is actually a pointer to the first item of the object
</span><span> </span><span style="color:#608b4e;">// so str+m is the address of the m-ieth item of the object
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%c </span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>(str+m));
</span><span> m++;
</span><span>}
</span></code></pre>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#608b4e;">// array of pointers to characters
</span><span style="color:#569cd6;">char *</span><span>names[] = {</span><span style="color:#d69d85;">"Alice"</span><span>, </span><span style="color:#d69d85;">"Bob"</span><span>, </span><span style="color:#d69d85;">"Charlie"</span><span>, </span><span style="color:#d69d85;">"David"</span><span>, </span><span style="color:#d69d85;">"Emma"</span><span>};
</span><span style="color:#608b4e;">// Pointer to the first element of the first element of the array of strings
</span><span style="color:#569cd6;">char **</span><span>ptr_names = names;
</span><span>
</span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> j = </span><span style="color:#b5cea8;">0</span><span>; j < </span><span style="color:#b5cea8;">5</span><span>; j++) {
</span><span> </span><span style="color:#608b4e;">// Print elements of the array of strings using indices
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, names[j]);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> x = </span><span style="color:#b5cea8;">0</span><span>; x < </span><span style="color:#b5cea8;">5</span><span>; x++) {
</span><span> </span><span style="color:#608b4e;">// Print the element currently pointed to
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>ptr_names);
</span><span> ptr_names = ptr_names + </span><span style="color:#b5cea8;">1</span><span>;
</span><span>}
</span></code></pre>
<h3 id="p">Passing arguments by reference:</h3>
<p>C Pointers can be used to pass arguments by reference to functions, allowing the function to modify the value of the original variable. This is particularly useful in the following situations:</p>
<ul>
<li>working with large data structures</li>
<li>you need to modify the value of a variable in a function</li>
<li>you need to optimize code by reducing the number of copies of data that need to be made</li>
<li>you need to return more than one value</li>
</ul>
<p>Example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><string.h>
</span><span>
</span><span style="color:#608b4e;">// let's suppose this is a laaaaaarge struct
</span><span style="color:#569cd6;">typedef struct </span><span>{
</span><span> </span><span style="color:#569cd6;">int</span><span> id;
</span><span> </span><span style="color:#569cd6;">char</span><span> name[</span><span style="color:#b5cea8;">50</span><span>];
</span><span> </span><span style="color:#569cd6;">float</span><span> salary;
</span><span>} </span><span style="color:#4ec9b0;">Employee</span><span>;
</span><span>
</span><span>
</span><span style="color:#569cd6;">void </span><span>update_employee(Employee</span><span style="color:#569cd6;">* </span><span>emp) {
</span><span> emp->salary *= </span><span style="color:#b5cea8;">1.1</span><span>; </span><span style="color:#608b4e;">// 10% raise
</span><span>}
</span><span>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> </span><span style="color:#608b4e;">// We create the large structure once
</span><span> Employee e1 = {</span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">"John Doe"</span><span>, </span><span style="color:#b5cea8;">5000</span><span>};
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Employee: </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">, Salary: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, e1.name, e1.salary);
</span><span> </span><span style="color:#608b4e;">// We pass the large structure by reference
</span><span> </span><span style="color:#608b4e;">// to avoid the overhead of copying the entire structure.
</span><span> update_employee(</span><span style="color:#569cd6;">&</span><span>e1);
</span><span> printf(</span><span style="color:#d69d85;">"Employee after update: </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">, Salary: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, e1.name, e1.salary);
</span><span>
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span><span>
</span></code></pre>
<h3 id="m">Allocating memory:</h3>
<p>C Pointers are also used to allocate memory dynamically at run-time using functions like <a href="https://man7.org/linux/man-pages/man3/malloc.3.html">malloc()</a>, <a href="https://man7.org/linux/man-pages/man3/calloc.3p.html">calloc()</a>, and <a href="https://man7.org/linux/man-pages/man3/realloc.3p.html">realloc()</a>. Examples:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><string.h>
</span><span>
</span><span style="color:#608b4e;">// a huuuuuuuge structure
</span><span style="color:#569cd6;">typedef struct </span><span>{
</span><span> </span><span style="color:#569cd6;">int</span><span> id;
</span><span> </span><span style="color:#569cd6;">char</span><span> name[</span><span style="color:#b5cea8;">50</span><span>];
</span><span> </span><span style="color:#569cd6;">float</span><span> salary;
</span><span>} </span><span style="color:#4ec9b0;">Employee</span><span>;
</span><span>
</span><span style="color:#608b4e;">// thanks to pointer & to malloc,
</span><span style="color:#608b4e;">// we can create the structure inside the create_employee function
</span><span style="color:#608b4e;">// and use the structure even after the function goes out of scope
</span><span>Employee</span><span style="color:#569cd6;">* </span><span>create_employee(){
</span><span> </span><span style="color:#608b4e;">// Allocate memory space on heap, size should be the same as width of Employee struct
</span><span> </span><span style="color:#608b4e;">// Allocate a pointer on stack,
</span><span> Employee</span><span style="color:#569cd6;">*</span><span> e2 = malloc(</span><span style="color:#569cd6;">sizeof</span><span>(Employee));
</span><span> </span><span style="color:#608b4e;">// always verify the allocation worked fine
</span><span> </span><span style="color:#569cd6;">if </span><span>(e2 == nullptr) {
</span><span> printf(</span><span style="color:#d69d85;">"Memory allocation failed!</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> exit(EXIT_FAILURE);
</span><span> }
</span><span> </span><span style="color:#608b4e;">// the ptr e2 contains the value of the memory address of the struct that is on the heap
</span><span> </span><span style="color:#569cd6;">return</span><span> e2;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>init_employee(Employee</span><span style="color:#569cd6;">* </span><span>emp, </span><span style="color:#569cd6;">int </span><span>id, </span><span style="color:#569cd6;">const char* </span><span>name, </span><span style="color:#569cd6;">float </span><span>salary) {
</span><span> emp->id = id;
</span><span> strcpy(emp->name, name);
</span><span> emp->salary = salary;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>update_employee(Employee</span><span style="color:#569cd6;">* </span><span>emp) {
</span><span> emp->salary *= </span><span style="color:#b5cea8;">1.1</span><span>; </span><span style="color:#608b4e;">// 10% raise
</span><span>}
</span><span>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> Employee</span><span style="color:#569cd6;">*</span><span> e2 = create_employee();
</span><span> init_employee(e2, </span><span style="color:#b5cea8;">2</span><span>, </span><span style="color:#d69d85;">"Nsukami Patrick"</span><span>, </span><span style="color:#b5cea8;">6000</span><span>);
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Employee: </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">, Salary: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, e2->name, e2->salary);
</span><span> update_employee(e2);
</span><span> printf(</span><span style="color:#d69d85;">"Employee after update: </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">, Salary: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, e2->name, e2->salary);
</span><span>
</span><span> </span><span style="color:#608b4e;">// never forget to free the allocated memory
</span><span> free(e2);
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<h3 id="s">Implementing data structures:</h3>
<p><a href="#">Linked lists</a>, <a href="#">Stacks</a>, <a href="#">Queues</a>, <a href="#">Trees</a>, <a href="#">Graphs</a> and many other data structures are implemented using C Pointers. Example:</p>
<h3 id="f">Calling functions:</h3>
<p>A function pointer, is a pointer which point to the memory address of a function instead of pointing to the address of a data object. They are used to:</p>
<ul>
<li>store functions in data structures</li>
<li>pass functions as arguments to other functions</li>
<li>call functions dynamically at runtime</li>
</ul>
<p>Here are some <a href="https://nskm.xyz/posts/fp/">examples</a> on how to use function pointers in C.</p>
<h3 id="h">Interfacing with hardware:</h3>
<p>C Pointers are often used in low-level programming to interact with hardware devices and memory-mapped registers. They provide a way to access specific memory addresses and manipulate hardware directly. This part goes way beyond what my brain is able to understand. Hopefully, I'll be able to show you an example in another article.</p>
<h2 id="g">Going further:</h2>
<p>To learn more about C pointers, we recommend the following <strong>pointers</strong>:</p>
<ul>
<li><a href="https://i.imgur.com/SIN2jJm.png">0x7ffc6155fd68</a></li>
<li><a href="https://en.wikipedia.org/wiki/Pointer_(dog_breed)">0x7ffc6155fd69</a></li>
<li><a href="https://en.wikipedia.org/wiki/Pointer_(computer_programming)">0x7ffc6155fd70</a></li>
<li><a href="https://yewtu.be/playlist?list=PLdo5W4Nhv31a8UcMN9-35ghv8qyFWD9_S">0x7ffc6155fd71</a></li>
<li><a href="https://man7.org/linux/man-pages/man1/setleds.1.html">0x7ffc6155fd72</a></li>
<li><a href="https://www.adafruit.com/product/2264">0x7ffc6155fd73</a></li>
</ul>
<br>
<img style="display: block; margin: 0 auto; width: 550px" src="/assets/ptrs.jpg" alt="Pointers" title="Pointers"/>
C2023-09-25T00:00:00+00:002023-09-25T00:00:00+00:00
Unknown
https://nskm.xyz/posts/cp/<p>I spent some time adding <a href="https://www.logilab.fr/blogentry/13884378">type annotations</a> to some Python code. Shoutout to all the people at <a href="https://logilab.fr">Logilab</a>, a really awesome team doing really awesome things. All those type annotations made me more and more interested in static typing... and in C programming.</p>
<p><em><strong>Target audience</strong></em>: people who already have an experience with programming. People who may have written some C programs in the past and who want to refresh their memory.</p>
<p><em><strong>Context</strong></em>: I'm working on:</p>
<ul>
<li>I'm working on <code>Linux 5.13.0-52-generic #59-Ubuntu SMP Wed Jun 15 20:17:13 UTC 2022 x86_64</code></li>
<li>And I'm using <code>GCC (Ubuntu 11.2.0-7ubuntu2) 11.2.0</code></li>
</ul>
<p><a href="https://yewtu.be/watch?v=6YSNKq-6SjM">Music please</a></p>
<img style="display: block; margin: 0 auto; width: 600px" src="/assets/force.jpg" alt="Force" title="Force"/>
<br>
<br>
<h2 id="toc">ToC</h2>
<ul>
<li><a href="https://nskm.xyz/posts/cp/#setup">Setup</a></li>
<li><a href="https://nskm.xyz/posts/cp/#primitive">Primitive types</a>
<ul>
<li><a href="https://nskm.xyz/posts/cp/#integers">integers</a></li>
<li><a href="https://nskm.xyz/posts/cp/#floats">floats</a></li>
<li><a href="https://nskm.xyz/posts/cp/#bool">booleans</a></li>
<li><a href="https://nskm.xyz/posts/cp/#void">void</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/cp/#composite">Composite types</a>
<ul>
<li><a href="https://nskm.xyz/posts/cp/#structs">structs</a></li>
<li><a href="https://nskm.xyz/posts/cp/#enums">enums</a></li>
<li><a href="https://nskm.xyz/posts/cp/#unions">unions</a></li>
<li><a href="https://nskm.xyz/posts/cp/#arrays">arrays</a></li>
<li><a href="https://nskm.xyz/posts/cp/#strings">strings</a></li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/cp/#constants">Constants</a></li>
</ul>
<h2 id="setup">The setup:</h2>
<p>To write C programs, we need:</p>
<ul>
<li>An <a href="#">operating system</a> that respect your freedom and privacy.</li>
<li>A C compiler such as <a href="https://gcc.gnu.org/">gcc</a> or <a href="https://clang.llvm.org/">clang</a>.</li>
<li>A <a href="https://www.gnu.org/software/emacs/">text editor</a> that <a href="/images/trollface.jpg/"><em>respects</em></a> your freedom and privacy.</li>
<li>(Optional) a Bash script to generate the project <a href="/snips/scfldc/">boilerplate</a>.</li>
<li>(Optional) a <a href="/snips/scfldc/">Makefile</a> to format, build, install and run the program quickly.</li>
</ul>
<br>
<h2 id="types">Types:</h2>
<p>Variable, a named location in the computer's memory where data can be stored and accessed. Type, define the size, the format, the behavior, the operations that can be performed on the variable. Let's talk about the common types in C programming.</p>
<h2 id="primitive">Primitive types:</h2>
<p>Primitive data types, or "basic data types" or "fundamental data types" are the most basic data types that can be used for representing simple values such as numbers, characters, etc... The primitive data types are...</p>
<h3 id="integers">Integer types</h3>
<ul>
<li>char: represents a single character (usually 1 byte)</li>
<li>short: represents a small integer</li>
<li>int: represents an integer</li>
<li>long: represents a long integer</li>
<li>long long: represents a very long integer</li>
<li>unsigned short: represent a small positive integer</li>
<li>unsigned int: represents a positive integer</li>
<li>unsigned long: represents a long positive integer</li>
<li>unsigned long long: represents a very long positive integer</li>
</ul>
<p>This list is not exhaustive, there is also what we call <strong>fixed width integer types</strong> and what we call <strong>minimum width integer types</strong>. We won't talk about them here, please browse this <a href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/stdint.h.html">link</a> for reference.</p>
<p>To find the range of a particular type, we can look for the <code>limits.h</code> file <a href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/limits.h.html">online</a>. Or, we can also look for the same file inside our computer:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>lim (trunk </span><span style="color:#569cd6;">*</span><span>%) >> find /usr -type f -name </span><span style="color:#d69d85;">"limits.h"
</span><span>/usr/src/linux-headers-5.13.0-51/include/linux/limits.h
</span><span>/usr/src/linux-headers-5.13.0-51/include/uapi/linux/limits.h
</span><span>/usr/src/linux-headers-5.13.0-51/include/vdso/limits.h
</span><span>/usr/src/linux-headers-5.13.0-52/include/linux/limits.h
</span><span>/usr/src/linux-headers-5.13.0-52/include/uapi/linux/limits.h
</span><span>/usr/src/linux-headers-5.13.0-52/include/vdso/limits.h
</span><span>/usr/include/linux/limits.h
</span><span>/usr/include/c++/11/tr1/limits.h
</span><span>/usr/include/limits.h
</span><span>/usr/lib/gcc/x86_64-linux-gnu/11/include/limits.h
</span><span>/usr/lib/x86_64-linux-gnu/perl5/5.32/Tk/pTk/compat/limits.h
</span><span>/usr/lib/llvm-13/lib/clang/13.0.0/include/limits.h
</span><span>lim (trunk </span><span style="color:#569cd6;">*</span><span>%) >>
</span></code></pre>
<p>If we open, let's say, the file <code>/usr/include/limits.h</code>, we can see interesting informations like :</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;"># define CHAR_MAX UCHAR_MAX
</span><span style="color:#9b9b9b;"># else
</span><span style="color:#9b9b9b;"># define CHAR_MIN SCHAR_MIN
</span><span style="color:#9b9b9b;"># define CHAR_MAX SCHAR_MAX
</span><span style="color:#9b9b9b;"># endif
</span><span>
</span><span style="color:#608b4e;">/* Minimum and maximum values a `signed short int' can hold. */
</span><span style="color:#9b9b9b;"># define SHRT_MIN (</span><span>-</span><span style="color:#b5cea8;">32768</span><span style="color:#9b9b9b;">)
</span><span style="color:#9b9b9b;"># define SHRT_MAX </span><span style="color:#b5cea8;">32767
</span><span>
</span><span style="color:#608b4e;">/* Maximum value an `unsigned short int' can hold. (Minimum is 0.) */
</span><span style="color:#9b9b9b;"># define USHRT_MAX </span><span style="color:#b5cea8;">65535
</span><span>
</span><span style="color:#608b4e;">/* Minimum and maximum values a `signed int' can hold. */
</span><span style="color:#9b9b9b;"># define INT_MIN (</span><span>-</span><span style="color:#9b9b9b;">INT_MAX </span><span>- </span><span style="color:#b5cea8;">1</span><span style="color:#9b9b9b;">)
</span><span style="color:#9b9b9b;"># define INT_MAX </span><span style="color:#b5cea8;">2147483647
</span><span>
</span><span style="color:#608b4e;">/* Maximum value an `unsigned int' can hold. (Minimum is 0.) */
</span><span style="color:#9b9b9b;"># define UINT_MAX </span><span style="color:#b5cea8;">4294967295</span><span style="color:#569cd6;">U
</span><span>
</span><span style="color:#608b4e;">/* Minimum and maximum values a `signed long int' can hold. */
</span><span style="color:#9b9b9b;"># if __WORDSIZE == 64
</span><span style="color:#9b9b9b;"># define LONG_MAX </span><span style="color:#b5cea8;">9223372036854775807</span><span style="color:#569cd6;">L
</span><span style="color:#9b9b9b;"># else
</span><span style="color:#9b9b9b;"># define LONG_MAX </span><span style="color:#b5cea8;">2147483647</span><span style="color:#569cd6;">L
</span></code></pre>
<h3 id="floats">Floating-point types:</h3>
<p>Used to represent real numbers with decimal points. More informations <a href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/float.h.html">here</a></p>
<ul>
<li>float: represents a single-precision floating-point number</li>
<li>double: represents a double-precision floating-point number</li>
<li>long double: represents an extended-precision floating-point number</li>
</ul>
<p>Here is an example showing how to use float values:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> </span><span style="color:#569cd6;">char</span><span> input[</span><span style="color:#b5cea8;">100</span><span>];
</span><span> </span><span style="color:#569cd6;">float</span><span> radius, area;
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Enter the radius of the circle: "</span><span>);
</span><span> </span><span style="color:#569cd6;">if</span><span>(fgets(input, </span><span style="color:#569cd6;">sizeof</span><span> input, stdin) != </span><span style="color:#569cd6;">NULL</span><span>) {
</span><span> radius = strtof(input, </span><span style="color:#569cd6;">NULL</span><span>);
</span><span> area = </span><span style="color:#b5cea8;">3.14 </span><span style="color:#569cd6;">*</span><span> radius </span><span style="color:#569cd6;">*</span><span> radius;
</span><span> printf(</span><span style="color:#d69d85;">"Area of the circle: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, area);
</span><span> }
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<p>I have found the file containing all the constants definitions for float values, but I am not (yet) able to correctly understand what's going on in this header file:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>lim (trunk </span><span style="color:#569cd6;">*</span><span>%) >> find /usr -type f -name </span><span style="color:#d69d85;">"float.h"
</span><span>/usr/include/tcl8.6/tcl-private/compat/float.h
</span><span>/usr/include/c++/11/tr1/float.h
</span><span>/usr/lib/gcc/x86_64-linux-gnu/11/include/float.h
</span><span>/usr/lib/llvm-13/lib/clang/13.0.0/include/float.h
</span><span>lim (trunk </span><span style="color:#569cd6;">*</span><span>%) >>
</span></code></pre>
<p>Here is a snippet of things that may be of interest for anyone looking for the ranges:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#if </span><span style="color:#569cd6;">defined</span><span style="color:#9b9b9b;"> __STDC_VERSION__ && __STDC_VERSION__ > 201710L
</span><span style="color:#608b4e;">/* Maximum finite positive value with MANT_DIG digits in the
</span><span style="color:#608b4e;"> significand taking their maximum value. */
</span><span style="color:#9b9b9b;">#undef FLT_NORM_MAX
</span><span style="color:#9b9b9b;">#undef DBL_NORM_MAX
</span><span style="color:#9b9b9b;">#undef LDBL_NORM_MAX
</span><span style="color:#9b9b9b;">#define FLT_NORM_MAX __FLT_NORM_MAX__
</span><span style="color:#9b9b9b;">#define DBL_NORM_MAX __DBL_NORM_MAX__
</span><span style="color:#9b9b9b;">#define LDBL_NORM_MAX __LDBL_NORM_MAX__
</span><span>
</span><span style="color:#608b4e;">/* Whether each type matches an IEC 60559 format (1 for format, 2 for
</span><span style="color:#608b4e;"> format and operations). */
</span><span style="color:#9b9b9b;">#undef FLT_IS_IEC_60559
</span><span style="color:#9b9b9b;">#undef DBL_IS_IEC_60559
</span><span style="color:#9b9b9b;">#undef LDBL_IS_IEC_60559
</span><span style="color:#9b9b9b;">#define FLT_IS_IEC_60559 __FLT_IS_IEC_60559__
</span><span style="color:#9b9b9b;">#define DBL_IS_IEC_60559 __DBL_IS_IEC_60559__
</span><span style="color:#9b9b9b;">#define LDBL_IS_IEC_60559 __LDBL_IS_IEC_60559__
</span></code></pre>
<h3 id="bool">Boolean type:</h3>
<p>The data type that can have two possible values: true or false. Include the <a href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/stdbool.h.html">"stdbool.h"</a> header file in your program to use this data type. Example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdbool.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><string.h>
</span><span>
</span><span style="color:#569cd6;">bool </span><span>strtobool(</span><span style="color:#569cd6;">const char* </span><span>str) {
</span><span> </span><span style="color:#569cd6;">if </span><span>(strcmp(str, </span><span style="color:#d69d85;">"true"</span><span>) == </span><span style="color:#b5cea8;">0 </span><span>|| strcmp(str, </span><span style="color:#d69d85;">"1"</span><span>) == </span><span style="color:#b5cea8;">0</span><span>) {
</span><span> </span><span style="color:#569cd6;">return true</span><span>;
</span><span> } </span><span style="color:#569cd6;">else if </span><span>(strcmp(str, </span><span style="color:#d69d85;">"false"</span><span>) == </span><span style="color:#b5cea8;">0 </span><span>|| strcmp(str, </span><span style="color:#d69d85;">"0"</span><span>) == </span><span style="color:#b5cea8;">0</span><span>) {
</span><span> </span><span style="color:#569cd6;">return false</span><span>;
</span><span> } </span><span style="color:#569cd6;">else </span><span>{
</span><span> fprintf(stderr, </span><span style="color:#d69d85;">"Invalid input: </span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, str);
</span><span> exit(EXIT_FAILURE);
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> </span><span style="color:#569cd6;">char</span><span> name[</span><span style="color:#b5cea8;">100</span><span>];
</span><span> </span><span style="color:#569cd6;">char</span><span> answer[</span><span style="color:#b5cea8;">5</span><span>];
</span><span> </span><span style="color:#569cd6;">bool</span><span> is_learning;
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Enter your name: "</span><span>);
</span><span> </span><span style="color:#569cd6;">if</span><span>(fgets(name, </span><span style="color:#569cd6;">sizeof</span><span> name, stdin) != </span><span style="color:#569cd6;">NULL</span><span>) {
</span><span> printf(</span><span style="color:#d69d85;">"Your name is </span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, name);
</span><span> }
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Are you learning C programming: "</span><span>);
</span><span> </span><span style="color:#569cd6;">if</span><span>(fgets(answer, </span><span style="color:#569cd6;">sizeof</span><span> answer, stdin) != </span><span style="color:#569cd6;">NULL</span><span>) {
</span><span> is_learning = strtobool(answer);
</span><span> </span><span style="color:#569cd6;">if </span><span>(is_learning)
</span><span> printf(</span><span style="color:#d69d85;">"Very good, keep learning.</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> </span><span style="color:#569cd6;">else
</span><span> printf(</span><span style="color:#d69d85;">"You should learn C programming.</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> }
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<h3 id="void">Void type:</h3>
<p>Void is a keyword to use as a placeholder where you would put a data type, to represent "no data". The void data type represents the absence of type. Void types are used in the following situations:</p>
<ul>
<li>to state that the specific type of the variable is not known <em>yet</em> (void pointers).</li>
<li>to state that the function returns no result (void functions).</li>
<li>to state that the function has no parameters.</li>
</ul>
<blockquote>
<p><strong>IMPORTANT: We can't directly dereference void pointers because the compiler doesn't know the size or type of data being pointed to. We need to explicitly cast them to the correct type before dereferencing.</strong></p>
</blockquote>
<p>Here's an example to illustrate the usage of void type:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#608b4e;">// greet is a function that
</span><span style="color:#608b4e;">// returns no result
</span><span style="color:#608b4e;">// and takes an unkown/undefined list of arguments
</span><span style="color:#569cd6;">void </span><span>greet(){
</span><span> printf(</span><span style="color:#d69d85;">"Hello world</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>main(){
</span><span> </span><span style="color:#608b4e;">// notice how we can pass all the arguments we want
</span><span> greet(</span><span style="color:#d69d85;">"foo"</span><span>, </span><span style="color:#b5cea8;">3.14</span><span>, </span><span style="color:#569cd6;">true</span><span>, </span><span style="color:#b5cea8;">1</span><span>);
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<p>A function declared with the void data type can't return a value, we cannot use the return keyword inside a void function:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>cscripts >> gcc main.c </span><span style="color:#569cd6;">&& </span><span>./a.out
</span><span>main.c: In function ‘greet’:
</span><span>main.c:57:10: warning: ‘return’ with a value, in function returning void
</span><span> 57 </span><span style="color:#569cd6;">| return</span><span> 5432</span><span style="color:#569cd6;">;
</span><span> </span><span style="color:#569cd6;">| </span><span>^~~~
</span><span>main.c:56:6: note: declared here
</span><span> 56 </span><span style="color:#569cd6;">| </span><span>void greet(void){
</span><span> </span><span style="color:#569cd6;">| </span><span>^~~~~~
</span></code></pre>
<p>Another example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#608b4e;">// greet is a function that
</span><span style="color:#608b4e;">// returns an int
</span><span style="color:#608b4e;">// and takes no arguments
</span><span style="color:#569cd6;">int </span><span>greet(</span><span style="color:#569cd6;">void</span><span>){
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">5432</span><span>;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>main(){
</span><span> </span><span style="color:#608b4e;">// Because greet returns a result, I can retrieve and print it
</span><span> printf(</span><span style="color:#d69d85;">"Greet returned the value: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, greet());
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<p>If we try to pass an argument to the greet function, we will receive an error message:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>cscripts >> gcc main.c </span><span style="color:#569cd6;">&& </span><span>./a.out
</span><span>main.c: In function ‘main’:
</span><span>main.c:57:44: error: too many arguments to function ‘greet’
</span><span> 57 </span><span style="color:#569cd6;">| </span><span>printf(</span><span style="color:#d69d85;">"Greet returned the value: %d\n"</span><span>, greet(</span><span style="color:#d69d85;">"foo"</span><span>))</span><span style="color:#569cd6;">;
</span><span> </span><span style="color:#569cd6;">| </span><span>^~~~~
</span><span>main.c:52:5: note: declared here
</span><span> 52 </span><span style="color:#569cd6;">| </span><span>int greet(void){
</span><span> </span><span style="color:#569cd6;">| </span><span>^~~~~
</span></code></pre>
<p>Last example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#569cd6;">void </span><span>main(){
</span><span> </span><span style="color:#569cd6;">int</span><span> age = </span><span style="color:#b5cea8;">5432</span><span>;
</span><span> </span><span style="color:#569cd6;">char*</span><span> name = </span><span style="color:#d69d85;">"Nsukami_"</span><span>;
</span><span>
</span><span> </span><span style="color:#608b4e;">// foo is a pointer that point to a not yet knwon data type
</span><span> </span><span style="color:#569cd6;">void*</span><span> foo;
</span><span>
</span><span> </span><span style="color:#608b4e;">// let's point to an integer
</span><span> foo = </span><span style="color:#569cd6;">&</span><span>age;
</span><span> </span><span style="color:#608b4e;">// explicit cast to the correct type before dereferencing
</span><span> printf(</span><span style="color:#d69d85;">"age is </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>((</span><span style="color:#569cd6;">int*</span><span>)foo));
</span><span>
</span><span> </span><span style="color:#608b4e;">// let's point to something else, for ex, a string
</span><span> foo = </span><span style="color:#569cd6;">&</span><span>name;
</span><span> </span><span style="color:#608b4e;">// explicit cast to the correct type before dereferencing
</span><span> printf(</span><span style="color:#d69d85;">"Name is </span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">*</span><span>((</span><span style="color:#569cd6;">char**</span><span>)foo));
</span><span>
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<h2 id="composite">Composite types</h2>
<p>Any data type which can be constructed using the language's primitive data types and other composite types.</p>
<h3 id="structs">Structures:</h3>
<p>Structs (short for structure), a user-defined data type that allows us to group together related data items of different types. Those data items are known as members or fields.</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#608b4e;">// An address is something with 2 attributes
</span><span style="color:#608b4e;">// a city street AND an house number
</span><span style="color:#569cd6;">typedef struct</span><span> address {
</span><span> </span><span style="color:#569cd6;">char</span><span> citystreet[</span><span style="color:#b5cea8;">100</span><span>];
</span><span> </span><span style="color:#569cd6;">int</span><span> housenumber;
</span><span>} </span><span style="color:#4ec9b0;">Address</span><span>;
</span><span>
</span><span style="color:#569cd6;">void </span><span>main() {
</span><span> Address my_address = {
</span><span> </span><span style="color:#d69d85;">"Planet 42, Rue de la Liberté"</span><span>,
</span><span> </span><span style="color:#b5cea8;">5432</span><span>,
</span><span> };
</span><span> printf(</span><span style="color:#d69d85;">"I live in </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">. My house number is: </span><span style="color:#b4cea8;">%u</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, my_address.citystreet, my_address.housenumber);
</span><span>
</span><span> </span><span style="color:#608b4e;">// how to update one of the fields
</span><span> my_address.housenumber = </span><span style="color:#b5cea8;">443</span><span>;
</span><span> printf(</span><span style="color:#d69d85;">"I live in </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">. My house number is now: </span><span style="color:#b4cea8;">%u</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, my_address.citystreet, my_address.housenumber);
</span><span>}
</span></code></pre>
<h3 id="enums">Enum data type:</h3>
<p>An enumeration is a <em>special kind</em> of data type defined by the user. An enumeration consists of a set of constant <em>integers</em> that are named by the user.</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span style="color:#569cd6;">typedef enum</span><span> Level {
</span><span> LOW = </span><span style="color:#b5cea8;">1</span><span>,
</span><span> MEDIUM,
</span><span> HIGH
</span><span>} </span><span style="color:#4ec9b0;">Level</span><span>;
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> Level level = MEDIUM;
</span><span>
</span><span> </span><span style="color:#569cd6;">switch </span><span>(level) {
</span><span> </span><span style="color:#569cd6;">case </span><span style="color:#b5cea8;">1</span><span>:
</span><span> printf(</span><span style="color:#d69d85;">"Low level"</span><span>);
</span><span> </span><span style="color:#569cd6;">break</span><span>;
</span><span> </span><span style="color:#569cd6;">case </span><span style="color:#b5cea8;">2</span><span>:
</span><span> printf(</span><span style="color:#d69d85;">"Medium level"</span><span>);
</span><span> </span><span style="color:#569cd6;">break</span><span>;
</span><span> </span><span style="color:#569cd6;">case </span><span style="color:#b5cea8;">3</span><span>:
</span><span> printf(</span><span style="color:#d69d85;">"High level"</span><span>);
</span><span> </span><span style="color:#569cd6;">break</span><span>;
</span><span> </span><span style="color:#569cd6;">default</span><span>:
</span><span> printf(</span><span style="color:#d69d85;">"Unkown level value"</span><span>);
</span><span> }
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<p>We said earlier an enumeration is a <em>special kind</em> of data type because.</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span>
</span></code></pre>
<h3 id="unions">Unions:</h3>
<p>An union is a user-defined data type that allows different data types to be stored in the same memory location. It enables us to create a variable that can hold different types of data, but only one type at a time.</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span style="color:#608b4e;">// Struct that combines an enum and a union
</span><span style="color:#569cd6;">typedef struct </span><span>{
</span><span> </span><span style="color:#608b4e;">// The type of our object could be either INT, either FLOAT, either STRING
</span><span> </span><span style="color:#569cd6;">enum </span><span>{
</span><span> INT,
</span><span> FLOAT,
</span><span> STRING
</span><span> } type;
</span><span>
</span><span> </span><span style="color:#608b4e;">// all 3 values share the same memory location, only one can be set
</span><span> </span><span style="color:#608b4e;">// value can be a int OR a float OR a string
</span><span> </span><span style="color:#569cd6;">union </span><span>{
</span><span> </span><span style="color:#569cd6;">int</span><span> intValue;
</span><span> </span><span style="color:#569cd6;">float</span><span> floatValue;
</span><span> </span><span style="color:#569cd6;">char</span><span> stringValue[</span><span style="color:#b5cea8;">20</span><span>];
</span><span> } value;
</span><span>} </span><span style="color:#4ec9b0;">Object</span><span>;
</span><span>
</span><span style="color:#608b4e;">// Function to print the object based on its type
</span><span style="color:#569cd6;">void </span><span>print(Object o) {
</span><span> </span><span style="color:#608b4e;">// check the type of our object
</span><span> </span><span style="color:#569cd6;">switch</span><span>(o.type) {
</span><span> </span><span style="color:#569cd6;">case </span><span>INT:
</span><span> </span><span style="color:#608b4e;">// then access the correct value
</span><span> printf(</span><span style="color:#d69d85;">"Integer: </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, o.value.intValue);
</span><span> </span><span style="color:#569cd6;">break</span><span>;
</span><span> </span><span style="color:#569cd6;">case </span><span>FLOAT:
</span><span> printf(</span><span style="color:#d69d85;">"Float: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, o.value.floatValue);
</span><span> </span><span style="color:#569cd6;">break</span><span>;
</span><span> </span><span style="color:#569cd6;">case</span><span> STRING:
</span><span> printf(</span><span style="color:#d69d85;">"String: </span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, o.value.stringValue);
</span><span> </span><span style="color:#569cd6;">break</span><span>;
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> </span><span style="color:#608b4e;">// Creating objects of different types
</span><span> Object o1, o2, o3;
</span><span>
</span><span> o1.type = INT;
</span><span> o1.value.intValue = </span><span style="color:#b5cea8;">10</span><span>;
</span><span>
</span><span> o2.type = FLOAT;
</span><span> o2.value.floatValue = </span><span style="color:#b5cea8;">3.14</span><span>;
</span><span>
</span><span> o3.type = STRING;
</span><span> strcpy(o3.value.stringValue, </span><span style="color:#d69d85;">"Hello"</span><span>);
</span><span>
</span><span> </span><span style="color:#608b4e;">// Printing the data objects
</span><span> print(o1);
</span><span> print(o2);
</span><span> print(o3);
</span><span>
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<h3 id="arrays">Arrays:</h3>
<p>A type consisting of nonempty items of the same nature stored using contiguous memory location. The number of items (the size) never changes during the array lifetime. Example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> </span><span style="color:#569cd6;">float</span><span> numbers[] = {</span><span style="color:#b5cea8;">2.5</span><span>, </span><span style="color:#b5cea8;">3.8</span><span>, </span><span style="color:#b5cea8;">4.2</span><span>, </span><span style="color:#b5cea8;">1.9</span><span>, </span><span style="color:#b5cea8;">5.6</span><span>};
</span><span> </span><span style="color:#569cd6;">int</span><span> size = </span><span style="color:#569cd6;">sizeof</span><span>(numbers) / </span><span style="color:#569cd6;">sizeof</span><span>(numbers[</span><span style="color:#b5cea8;">0</span><span>]);
</span><span> </span><span style="color:#569cd6;">float</span><span> sum = </span><span style="color:#b5cea8;">0.0</span><span>, average;
</span><span>
</span><span> </span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> i = </span><span style="color:#b5cea8;">0</span><span>; i < size; i++) {
</span><span> sum += numbers[i];
</span><span> }
</span><span>
</span><span> average = sum / size;
</span><span> printf(</span><span style="color:#d69d85;">"Average: </span><span style="color:#b4cea8;">%.2f</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, average);
</span><span>
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<h3 id="strings">Strings:</h3>
<p>A string is just a sequence of characters stored inside an array and ending with the null character: '\0'. We don't have strings as we have them in Python programming.</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> </span><span style="color:#569cd6;">char</span><span> name[</span><span style="color:#b5cea8;">100</span><span>];
</span><span> printf(</span><span style="color:#d69d85;">"Enter your name: "</span><span>);
</span><span>
</span><span> </span><span style="color:#569cd6;">if</span><span>(fgets(name, </span><span style="color:#569cd6;">sizeof</span><span> name, stdin) != </span><span style="color:#569cd6;">NULL</span><span>) {
</span><span> printf(</span><span style="color:#d69d85;">"Your name is </span><span style="color:#b4cea8;">%s</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, name);
</span><span> }
</span><span> exit(EXIT_SUCCESS);
</span><span>}
</span></code></pre>
<p>To know what are the functions available for string manipulation, Python developers will type <code>help(str)</code>:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>Help on </span><span style="color:#ff3333;">class </span><span>str </span><span style="color:#569cd6;">in </span><span>module builtins:
</span><span>
</span><span style="color:#569cd6;">class </span><span>str(</span><span style="color:#4ec9b0;">object</span><span>)
</span><span> | str(object=</span><span style="color:#d69d85;">''</span><span>) -> str
</span><span> | str(bytes_or_buffer[, encoding[, errors]]) -> str
</span><span> |
</span><span> | Create a new string object </span><span style="color:#9b9b9b;">from </span><span>the given object. If encoding </span><span style="color:#ff3333;">or
</span><span> | errors </span><span style="color:#569cd6;">is </span><span>specified, then the object must expose a data buffer
</span><span> | that will be decoded using the given encoding </span><span style="color:#569cd6;">and </span><span>error handler.
</span><span> | Otherwise, returns the result of object.__str__() (</span><span style="color:#569cd6;">if </span><span>defined)
</span><span> | </span><span style="color:#569cd6;">or </span><span>repr(object).
</span><span> | encoding defaults to sys.getdefaultencoding().
</span><span> | errors defaults to </span><span style="color:#d69d85;">'strict'</span><span>.
</span><span> |
</span><span> | Methods defined here:
</span><span> |
</span><span> | __add__(self, value, /)
</span><span> | Return self+value.
</span><span> |
</span><span> | __contains__(self, key, /)
</span><span> | Return bool(key </span><span style="color:#569cd6;">in </span><span>self).
</span><span> |
</span><span> | __eq__(self, value, /)
</span><span> | Return self==value.
</span><span> |
</span><span> | __format__(self, format_spec, /)
</span><span>:
</span></code></pre>
<p>C developers will look inside the <code><string.h></code> header <a href="https://pubs.opengroup.org/onlinepubs/009695399/basedefs/string.h.html">file</a>.</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>nskm2 >> find /usr/include -type f -name "string.h"
</span><span>/usr/include/linux/string.h
</span><span>/usr/include/tcl8.6/tcl-private/compat/string.h
</span><span>/usr/include/string.h
</span><span>nskm2 >>
</span></code></pre>
<p>They can also read the man pages, <code>man string</code>:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>STRING(3) Linux Programmer's Manual STRING(3)
</span><span>
</span><span>NAME
</span><span> stpcpy, strcasecmp, strcat, strchr, strcmp, strcoll, strcpy, strcspn, strdup, strfry, strlen, strncat,
</span><span> strncmp, strncpy, strncasecmp, strpbrk, strrchr, strsep, strspn, strstr, strtok, strxfrm, index, rindex
</span><span> - string operations
</span><span>
</span><span>SYNOPSIS
</span><span> #include <strings.h>
</span><span>
</span><span> int strcasecmp(const char *s1, const char *s2);
</span><span> Compare the strings s1 and s2 ignoring case.
</span><span>
</span><span> int strncasecmp(const char *s1, const char *s2, size_t n);
</span><span> Compare the first n bytes of the strings s1 and s2 ignoring case.
</span><span>
</span><span> char *index(const char *s, int c);
</span><span> Return a pointer to the first occurrence of the character c in the string s.
</span><span>
</span><span> char *rindex(const char *s, int c);
</span><span> Return a pointer to the last occurrence of the character c in the string s.
</span><span>
</span><span> #include <string.h>
</span><span>
</span><span> char *stpcpy(char *dest, const char *src);
</span><span> Copy a string from src to dest, returning a pointer to the end of the resulting string at dest.
</span><span> Manual page string(3) line 1 (press h for help or q to quit)
</span></code></pre>
<h2 id="constants">Constants:</h2>
<p>A fixed value that cannot be, <em>that should not</em> be changed during the execution of a program.</p>
<ol>
<li>If we have a fixed value that is used in calculations, we can define it as a constant to avoid hard-coding the value multiple times.</li>
</ol>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#608b4e;">// let's define PI as a constant
</span><span style="color:#569cd6;">const int</span><span> PI = </span><span style="color:#b5cea8;">3.14159</span><span>;
</span><span style="color:#569cd6;">int</span><span> radius = </span><span style="color:#b5cea8;">5</span><span>;
</span><span style="color:#608b4e;">// let's reuse our constant
</span><span style="color:#569cd6;">int</span><span> area = PI </span><span style="color:#569cd6;">*</span><span> radius </span><span style="color:#569cd6;">*</span><span> radius;
</span></code></pre>
<p>Trying to modify our constant will give us an error message:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>main.c: In function ‘main’:
</span><span>main.c:129:6: error: assignment of read-only variable ‘PI’
</span><span> 129 </span><span style="color:#569cd6;">| </span><span>PI = 4.5</span><span style="color:#569cd6;">;
</span><span> </span><span style="color:#569cd6;">| </span><span>^
</span></code></pre>
<ol start="2">
<li>Another way to use constants is this one. Instead of using arbitrary numbers directly in your code, you can define them as constants with meaningful names to improve code readability and maintainability.</li>
</ol>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#569cd6;">const int</span><span> LENGTH = </span><span style="color:#b5cea8;">3</span><span>;
</span><span style="color:#569cd6;">const int</span><span> MIN_AGE = </span><span style="color:#b5cea8;">18</span><span>;
</span><span style="color:#569cd6;">const int</span><span> MAX_AGE = </span><span style="color:#b5cea8;">50</span><span>;
</span><span>
</span><span style="color:#569cd6;">char</span><span> input[LENGTH];
</span><span style="color:#569cd6;">int</span><span> age;
</span><span>
</span><span>printf(</span><span style="color:#d69d85;">"Enter your age: "</span><span>);
</span><span style="color:#569cd6;">if</span><span>(fgets(input, </span><span style="color:#569cd6;">sizeof</span><span> input, stdin) != </span><span style="color:#569cd6;">NULL</span><span>) {
</span><span> age = strtol(input, </span><span style="color:#569cd6;">NULL</span><span>, </span><span style="color:#b5cea8;">10</span><span>);
</span><span> </span><span style="color:#569cd6;">if </span><span>(age < MIN_AGE || age > MAX_AGE) {
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d</span><span style="color:#d69d85;"> is an invalid age! Please enter an age between </span><span style="color:#b4cea8;">%d</span><span style="color:#d69d85;"> and </span><span style="color:#b4cea8;">%d</span><span style="color:#d69d85;">.</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, age, MIN_AGE, MAX_AGE);
</span><span> }
</span><span>}
</span></code></pre>
<ol start="3">
<li>When declaring an array, we can use a constant to specify its size, making it easier to modify if needed.</li>
</ol>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#608b4e;">// In this example, `ARRAY_SIZE` is declared as a constant to specify the size of an array. By using a
</span><span style="color:#608b4e;">// constant, you can easily change the size of the array by modifying the constant's value in a single place.
</span><span style="color:#569cd6;">const int</span><span> ARRAY_SIZE = </span><span style="color:#b5cea8;">10</span><span>;
</span><span style="color:#569cd6;">int</span><span> arr[ARRAY_SIZE];
</span></code></pre>
<ol start="4">
<li>We can also declare function parameters as <code>const</code> to ensure that the function does not modify the parameter:</li>
</ol>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#608b4e;">// This helps prevent accidental modifications and indicates that the function is read-only.
</span><span style="color:#569cd6;">void </span><span>print(</span><span style="color:#569cd6;">const int </span><span>arr[], </span><span style="color:#569cd6;">int </span><span>size) {
</span><span> </span><span style="color:#569cd6;">for </span><span>(</span><span style="color:#569cd6;">int</span><span> i = </span><span style="color:#b5cea8;">0</span><span>; i < size; i++) {
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d </span><span style="color:#d69d85;">"</span><span>, arr[i]);
</span><span> }
</span><span>}
</span></code></pre>
<h2 id="conclusion">Conclusion:</h2>
<p>I really hope you've learned something. I did \o/ In the next episode, I will try to write about header files or pointers or something else depending on the current moon, the weather and the stars alignment. Also, thanks to <a href="https://mastodon.social/@orbifx">Orbifx</a> for taking the time to answer all my questions.</p>
<h1 id="more-on-the-topic">More on the topic:</h1>
<ul>
<li><a href="https://hal.inria.fr/hal-02383654/file/ModernC.pdf">Modern C, Jens Gustedt</a></li>
<li><a href="https://www.gnu.org/software/gnu-c-manual/gnu-c-manual.pdf">The GNU C Reference Manual</a></li>
<li><a href="https://github.com/amjadmajid/Makefile">Explanation about how to write a Makefile</a></li>
<li><a href="https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2655.pdf">Make false and true first-class language features v4
proposal for C23</a></li>
</ul>
<h1 id="on-a-completely-differnt-note">On a completely differnt note:</h1>
<ul>
<li><a href="https://yewtu.be/watch?v=snvzAfYOFks">Des organismes aux algorithmes, l'odyssée computationnelle du vivant | Xavier Berthet | TEDxINPENSAT</a></li>
<li><a href="https://2023.djangocon.africa/">The first DjangoCon Africa, in Zanzibar, Tanzania.</a></li>
</ul>
Decorators2023-02-06T00:00:00+00:002023-02-06T00:00:00+00:00
Unknown
https://nskm.xyz/posts/decor/<p>I was watching a <em>really</em> interesting <a href="https://y.com.sb/watch?v=RL7BECNn-RI">video</a> about buildings architecture in New York, when one of my <a href="https://dit.sn">DIT</a> student asked me to explain to him what was happening in this <a href="https://git.disroot.org/nsukami/flask-user-authentication-example/src/branch/master/src/main/utils.py#L7">piece of code</a>. A function I have written few months ago to check for valid authentication tokens. In this post, I will try to show how decorators in <a href="https://python.org">Python</a>, a powerful concept, allows us to apply new functionality without modifying the existing structure of an object. <a href="https://y.com.sb/watch?v=cDk9L72ZkuE">Music please</a>.</p>
<img style="display: block; margin: 0 auto; width: 600px" src="/assets/decorated.png" alt="" title="Somewhere in Africa"/>
<br>
<br>
<h1 id="toc">TOC:</h1>
<ul>
<li><a href="https://nskm.xyz/posts/decor/#f">1 A function doing one thing</a></li>
<li><a href="https://nskm.xyz/posts/decor/#g">2 A decorator, modifying the behaviour of the function</a></li>
<li><a href="https://nskm.xyz/posts/decor/#h">3 A decorator should keep important informations</a></li>
<li><a href="https://nskm.xyz/posts/decor/#i">4 A decorator should receive a flexible number of argument</a></li>
<li><a href="https://nskm.xyz/posts/decor/#j">5 A decorator can also have parameters</a></li>
<li><a href="https://nskm.xyz/posts/decor/#k">6 We are still able to access the decorated function</a></li>
<li><a href="https://nskm.xyz/posts/decor/#l">7 An interesting example</a></li>
<li><a href="https://nskm.xyz/posts/decor/#m">8 Even more interesting, how decorators are used elsewhere</a></li>
</ul>
<br>
<h1 id="f">1. A function:</h1>
<p>Let's suppose we want to join 2 paths:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">2</span><span>]: </span><span style="color:#9b9b9b;">from </span><span>pathlib </span><span style="color:#9b9b9b;">import </span><span>Path
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">3</span><span>]: a = Path(</span><span style="color:#d69d85;">"foo"</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">4</span><span>]: a
</span><span>Out[</span><span style="color:#b5cea8;">4</span><span>]: PosixPath(</span><span style="color:#d69d85;">'foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">5</span><span>]: b = Path(</span><span style="color:#d69d85;">"bar"</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">6</span><span>]: a.joinpath(b)
</span><span>Out[</span><span style="color:#b5cea8;">6</span><span>]: PosixPath(</span><span style="color:#d69d85;">'foo/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">7</span><span>]:
</span></code></pre>
<p>Let's suppose we usually combine 2 paths together. So much that we created a function for that purpose:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">21</span><span>]: path_t = str | Path
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">22</span><span>]: </span><span style="color:#ff3333;">def </span><span>jp(path1: path_t, path2: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""Join 2 paths together"""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>isinstance(path1, str):
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(path1)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(path2)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">23</span><span>]: jp(b, a)
</span><span>Out[</span><span style="color:#b5cea8;">23</span><span>]: PosixPath(</span><span style="color:#d69d85;">'bar/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">24</span><span>]: jp(a, b)
</span><span>Out[</span><span style="color:#b5cea8;">24</span><span>]: PosixPath(</span><span style="color:#d69d85;">'foo/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">25</span><span>]: jp(</span><span style="color:#d69d85;">"corge"</span><span>, a)
</span><span>Out[</span><span style="color:#b5cea8;">25</span><span>]: PosixPath(</span><span style="color:#d69d85;">'corge/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">26</span><span>]: jp(</span><span style="color:#d69d85;">"corge"</span><span>, </span><span style="color:#d69d85;">"qux"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">26</span><span>]: PosixPath(</span><span style="color:#d69d85;">'corge/qux'</span><span>)
</span></code></pre>
<h1 id="g">2. A decorator modifying the behaviour of the function:</h1>
<p>While it may seem confusing at first, a decorator is really just <em>a function that takes a function and returns a function</em>. Let me explain.</p>
<p>Suppose we usually combine our home path with another path. Let's suppose the first parameter is often our home path. Let's suppose we can't modify the original function. Python functions are <a href="https://en.wikipedia.org/wiki/First-class_citizen">first-class citizens</a>. That means:</p>
<ul>
<li>We can pass a function as an argument of another function.</li>
<li>We can assign a function to a variable.</li>
<li>We can return a function as a value from another function.</li>
</ul>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">38</span><span>]: </span><span style="color:#ff3333;">def </span><span>homeprefix(func: Callable[[path_t, path_t], path_t]) -> Callable[[path_t], path_t]:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>wrapper(m: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A wrapper that takes one parameter."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>func(</span><span style="color:#d69d85;">"/home/nsukami"</span><span>, m)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>wrapper
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">39</span><span>]: f = homeprefix(jp)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">40</span><span>]: f(</span><span style="color:#d69d85;">"bar"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">40</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">41</span><span>]: f(b)
</span><span>Out[</span><span style="color:#b5cea8;">41</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">42</span><span>]: f(a)
</span><span>Out[</span><span style="color:#b5cea8;">42</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">43</span><span>]: f(</span><span style="color:#d69d85;">"corge/qux"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">43</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/corge/qux'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">44</span><span>]: f(Path(</span><span style="color:#d69d85;">"spam"</span><span>))
</span><span>Out[</span><span style="color:#b5cea8;">44</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/spam'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">45</span><span>]:
</span></code></pre>
<p>As strange as it may seems, we could have done the following:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">45</span><span>]: jp = homeprefix(jp) </span><span style="color:#608b4e;"># there is an alternative syntax
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">46</span><span>]: jp(a)
</span><span>Out[</span><span style="color:#b5cea8;">46</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">47</span><span>]: jp(b)
</span><span>Out[</span><span style="color:#b5cea8;">47</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">48</span><span>]: jp(</span><span style="color:#d69d85;">"foobar"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">48</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foobar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">49</span><span>]: jp(Path(</span><span style="color:#d69d85;">"corge"</span><span>))
</span><span>Out[</span><span style="color:#b5cea8;">49</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/corge'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">50</span><span>]:
</span></code></pre>
<p>Instead of <code>jp = homeprefix(jp)</code> we could use the following syntax which better vehicle our intent: <strong>modifying the behaviour of an existing function</strong>. Simply put: <strong>decorating</strong> our original function:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">50</span><span>]: </span><span style="color:#ff3333;">def </span><span>homeprefix(func):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>wrapper(m):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that takes one parameter."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>func(</span><span style="color:#d69d85;">"/home/nsukami"</span><span>, m)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>wrapper
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">51</span><span>]: </span><span style="color:#569cd6;">@</span><span>homeprefix </span><span style="color:#608b4e;"># here we are, decorating jp function
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>jp(path1: path_t, path2: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""Join 2 paths together"""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>isinstance(path1, str):
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(path1)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(path2)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">52</span><span>]: jp(Path(</span><span style="color:#d69d85;">"corge"</span><span>))
</span><span>Out[</span><span style="color:#b5cea8;">52</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/corge'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">53</span><span>]: jp(</span><span style="color:#d69d85;">"corge/qux"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">53</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/corge/qux'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">54</span><span>]: jp(a)
</span><span>Out[</span><span style="color:#b5cea8;">54</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">55</span><span>]: jp(b)
</span><span>Out[</span><span style="color:#b5cea8;">55</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">56</span><span>]:
</span></code></pre>
<h1 id="h">3. A decorator should keep important informations:</h1>
<p>Now we have another problem. Because we are literally replacing one function with another, we're also losing important informations like the docstring and the name:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">61</span><span>]: ?jp
</span><span>Signature: jp(m)
</span><span>Docstring: A function that takes one parameter.
</span><span>File: ~/GIT/nskm2/<ipython-input-</span><span style="color:#b5cea8;">59</span><span>-b4f67b1e800d>
</span><span>Type: function
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">62</span><span>]: jp.__name__
</span><span>Out[</span><span style="color:#b5cea8;">62</span><span>]: </span><span style="color:#d69d85;">'wrapper'
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">63</span><span>]:
</span></code></pre>
<p>That's the reason we have <a href="https://docs.python.org/3/library/functools.html#functools.wraps">functools.wraps</a>. <code>Wraps</code> takes the decorated function and adds the functionality of copying over the function name, docstring, arguments list, etc to <strong>the returned function</strong>. And since wraps is itself a decorator:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">63</span><span>]: </span><span style="color:#9b9b9b;">from </span><span>functools </span><span style="color:#9b9b9b;">import </span><span>wraps
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">64</span><span>]: </span><span style="color:#ff3333;">def </span><span>homeprefix(func):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">@</span><span>wraps(func)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>wrapper(m):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that takes one parameter."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>func(</span><span style="color:#d69d85;">"/home/nsukami"</span><span>, m)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>wrapper
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">65</span><span>]: </span><span style="color:#569cd6;">@</span><span>homeprefix
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>jp(path1: path_t, path2: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that join 2 paths together."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>isinstance(path1, str):
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(path1)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(path2)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">66</span><span>]: ?jp
</span><span>Signature: jp(path1: str | pathlib.Path, path2: str | pathlib.Path) -> str | pathlib.Path
</span><span>Docstring: A function that join </span><span style="color:#b5cea8;">2 </span><span>paths together.
</span><span>File: ~/GIT/nskm2/<ipython-input-</span><span style="color:#b5cea8;">65</span><span>-1703cd8a40dc>
</span><span>Type: function
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">67</span><span>]: jp.__name__
</span><span>Out[</span><span style="color:#b5cea8;">67</span><span>]: </span><span style="color:#d69d85;">'jp'
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">68</span><span>]:
</span></code></pre>
<h1 id="i">4. A decorator should receive a flexible number of argument:</h1>
<p>Now, let's suppose our original function can take many arguments:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">101</span><span>]: </span><span style="color:#ff3333;">def </span><span>jp(*args: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that join many paths together."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>args:
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(args[</span><span style="color:#b5cea8;">0</span><span>])
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(*args[</span><span style="color:#b5cea8;">1</span><span>:])
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">else</span><span>:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">raise </span><span>ValueError(</span><span style="color:#d69d85;">"args should be longer"</span><span>)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">102</span><span>]: jp(</span><span style="color:#d69d85;">"bac"</span><span>, </span><span style="color:#d69d85;">"abc"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">102</span><span>]: PosixPath(</span><span style="color:#d69d85;">'bac/abc'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">103</span><span>]: jp(</span><span style="color:#d69d85;">"bac"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">103</span><span>]: PosixPath(</span><span style="color:#d69d85;">'bac'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">104</span><span>]: jp(Path(</span><span style="color:#d69d85;">"abc"</span><span>), Path(</span><span style="color:#d69d85;">"foo"</span><span>), </span><span style="color:#d69d85;">"bar"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">104</span><span>]: PosixPath(</span><span style="color:#d69d85;">'abc/foo/bar'</span><span>)
</span></code></pre>
<p>The previous decorator is not flexible enough because the returned function can't take more than 1 parameter:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">106</span><span>]: </span><span style="color:#ff3333;">def </span><span>homeprefix(func):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">@</span><span>wraps(func)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>wrapper(m):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that takes one parameter."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>func(</span><span style="color:#d69d85;">"/home/nsukami"</span><span>, m)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>wrapper
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">107</span><span>]: </span><span style="color:#569cd6;">@</span><span>homeprefix
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>jp(*args: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that join many paths together."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>args:
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(args[</span><span style="color:#b5cea8;">0</span><span>])
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(*args[</span><span style="color:#b5cea8;">1</span><span>:])
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">else</span><span>:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">raise </span><span>ValueError(</span><span style="color:#d69d85;">"args should be longer"</span><span>)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">108</span><span>]: jp(</span><span style="color:#d69d85;">"a"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">108</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/a'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">109</span><span>]: jp(</span><span style="color:#d69d85;">"a"</span><span>, </span><span style="color:#d69d85;">"b"</span><span>)
</span><span>---------------------------------------------------------------------------
</span><span>TypeError Traceback (most recent call last)
</span><span>Cell In [</span><span style="color:#b5cea8;">109</span><span>], line </span><span style="color:#b5cea8;">1
</span><span>----> </span><span style="color:#b5cea8;">1 </span><span>jp(</span><span style="color:#d69d85;">"a"</span><span>, </span><span style="color:#d69d85;">"b"</span><span>)
</span><span>
</span><span>TypeError: jp() takes </span><span style="color:#b5cea8;">1 </span><span>positional argument but </span><span style="color:#b5cea8;">2 </span><span>were given
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">110</span><span>]:
</span><span>
</span></code></pre>
<p>Let's rewrite our decorator so that the returned function can take an infinte number or arguments:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">113</span><span>]: </span><span style="color:#ff3333;">def </span><span>homeprefix(func):
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">@</span><span>wraps(func)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>wrapper(*args): </span><span style="color:#608b4e;"># here is our update: using *args
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>func(</span><span style="color:#d69d85;">"/home/nsukami"</span><span>, </span><span style="color:#569cd6;">*</span><span>args)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>wrapper
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">114</span><span>]: </span><span style="color:#569cd6;">@</span><span>homeprefix
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>jp(*args: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that join many paths together."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>args:
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(args[</span><span style="color:#b5cea8;">0</span><span>])
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(*args[</span><span style="color:#b5cea8;">1</span><span>:])
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">else</span><span>:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">raise </span><span>ValueError(</span><span style="color:#d69d85;">"args should be longer"</span><span>)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">115</span><span>]: jp()
</span><span>Out[</span><span style="color:#b5cea8;">115</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">116</span><span>]: jp(</span><span style="color:#d69d85;">"b"</span><span>, </span><span style="color:#d69d85;">"c"</span><span>, </span><span style="color:#d69d85;">"d"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">116</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/b/c/d'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">117</span><span>]: jp(</span><span style="color:#d69d85;">"b"</span><span>, </span><span style="color:#d69d85;">"c"</span><span>, </span><span style="color:#d69d85;">"d"</span><span>, Path(</span><span style="color:#d69d85;">"foo"</span><span>))
</span><span>Out[</span><span style="color:#b5cea8;">117</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/b/c/d/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">118</span><span>]: jp(Path(</span><span style="color:#d69d85;">"foo"</span><span>), Path(</span><span style="color:#d69d85;">"bar"</span><span>))
</span><span>Out[</span><span style="color:#b5cea8;">118</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foo/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">119</span><span>]:
</span><span>
</span></code></pre>
<p>With the help of the special parameters *args and **kwargs, decorators become more generic. That way if we need the same kind of decorator in different functions with different arguments, we can write just one decorator. In general, a decorator is written like this:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>decorator(func):
</span><span> @wraps(func)
</span><span> </span><span style="color:#569cd6;">def </span><span>wrapper(</span><span style="color:#569cd6;">*</span><span>args, </span><span style="color:#569cd6;">**</span><span>kwargs):
</span><span> func(*args, </span><span style="color:#569cd6;">**</span><span>kwargs)
</span><span> </span><span style="color:#569cd6;">return </span><span>wrapper
</span></code></pre>
<h1 id="j">5. A decorator can also have parameters:</h1>
<p>Let's suppose people using the decorator want to be able to set what should the prefix used as the first parameter of our jp function. Let's suppose we want the decorator to receive one parameter, which is the prefix:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">70</span><span>]: </span><span style="color:#ff3333;">def </span><span>homeprefix(prefix): </span><span style="color:#608b4e;"># this function builds and returns a decorator
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>outer(func): </span><span style="color:#608b4e;"># this function is the decorator
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">@</span><span>wraps(func)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>inner(*args, </span><span style="color:#569cd6;">**</span><span>kwargs): </span><span style="color:#608b4e;"># the function wrapping the original function
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that takes one parameter."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>func(prefix, </span><span style="color:#569cd6;">*</span><span>args, </span><span style="color:#569cd6;">**</span><span>kwargs)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>inner
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>outer
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">71</span><span>]: </span><span style="color:#569cd6;">@</span><span>homeprefix(</span><span style="color:#d69d85;">"/home/nsukami"</span><span>)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>jp(path1: path_t, path2: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that join 2 paths together."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>isinstance(path1, str):
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(path1)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(path2)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">72</span><span>]: jp(a)
</span><span>Out[</span><span style="color:#b5cea8;">72</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">73</span><span>]: jp(b)
</span><span>Out[</span><span style="color:#b5cea8;">73</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/bar'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">74</span><span>]: jp(</span><span style="color:#d69d85;">"corge"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">74</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/corge'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">75</span><span>]:
</span></code></pre>
<p>We understand better what's going on when we do this instead:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">75</span><span>]: decorator = homeprefix(</span><span style="color:#d69d85;">"/home/patrick"</span><span>) </span><span style="color:#608b4e;"># setup a prefix
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">76</span><span>]: </span><span style="color:#569cd6;">@</span><span>decorator </span><span style="color:#608b4e;"># decorate
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#ff3333;">def </span><span>jp(path1: path_t, path2: path_t) -> path_t:
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#d69d85;">"""A function that join 2 paths together."""
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">if </span><span>isinstance(path1, str):
</span><span> </span><span style="color:#569cd6;">...</span><span>: path1 = Path(path1)
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>path1.joinpath(path2)
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">77</span><span>]: jp(</span><span style="color:#d69d85;">"corge"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">77</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/patrick/corge'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">78</span><span>]: jp(</span><span style="color:#d69d85;">"quux"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">78</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/patrick/quux'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">79</span><span>]:
</span></code></pre>
<h1 id="k">6. We are still able to access the decorated function:</h1>
<p>Let's suppose for some reason we want to test the function as if it was not decorated. Let's suppose we still want to use the underlying function. To achieve that, we'll use the <code>__wrapped__</code> attribute (assuming we used the <code>functools.wrap</code> function). <a href="https://docs.python.org/3/library/functools.html?highlight=__wrapped__">Somewhere</a> in the documentation, we can read:</p>
<blockquote>
<p>The original underlying function is accessible through the <strong>wrapped</strong> attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.</p>
</blockquote>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">127</span><span>]: jp(</span><span style="color:#d69d85;">"foo"</span><span>)
</span><span>Out[</span><span style="color:#b5cea8;">127</span><span>]: PosixPath(</span><span style="color:#d69d85;">'/home/nsukami/foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">128</span><span>]: jp.__wrapped__(</span><span style="color:#d69d85;">"foo"</span><span>) </span><span style="color:#608b4e;"># original function
</span><span>Out[</span><span style="color:#b5cea8;">128</span><span>]: PosixPath(</span><span style="color:#d69d85;">'foo'</span><span>)
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">129</span><span>]:
</span><span>
</span></code></pre>
<h1 id="l">7. An interesting example:</h1>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;">#!/usr/bin/env python3
</span><span>
</span><span style="color:#9b9b9b;">import </span><span>csv
</span><span style="color:#9b9b9b;">import </span><span>sqlite3
</span><span style="color:#9b9b9b;">import </span><span>pathlib
</span><span style="color:#9b9b9b;">from </span><span>functools </span><span style="color:#9b9b9b;">import </span><span>wraps
</span><span>
</span><span>here = pathlib.Path(__file__)
</span><span>db = here.parent.parent.parent / </span><span style="color:#d69d85;">"sqlite_stuff" </span><span>/ </span><span style="color:#d69d85;">"chinook" </span><span>/ </span><span style="color:#d69d85;">"chinook.db"
</span><span>
</span><span style="color:#569cd6;">def </span><span>connect(db):
</span><span> </span><span style="color:#608b4e;">"""A decorator to connect to an SQLite database and execute a query."""
</span><span> conn = sqlite3.connect(db)
</span><span> conn.isolation_level = </span><span style="color:#569cd6;">None
</span><span> </span><span style="color:#569cd6;">def </span><span>outer(f):
</span><span> @wraps(f)
</span><span> </span><span style="color:#569cd6;">def </span><span>inner(</span><span style="color:#569cd6;">*</span><span>args, </span><span style="color:#569cd6;">**</span><span>kwargs):
</span><span> query = f(*args, </span><span style="color:#569cd6;">**</span><span>kwargs)
</span><span> rset = conn.execute(query).fetchall()
</span><span> </span><span style="color:#569cd6;">return </span><span>rset
</span><span> </span><span style="color:#569cd6;">return </span><span>inner
</span><span> </span><span style="color:#569cd6;">return </span><span>outer
</span><span>
</span><span>
</span><span style="color:#569cd6;">def </span><span>columns(</span><span style="color:#569cd6;">*</span><span>columns, limit=</span><span style="color:#b5cea8;">10</span><span>):
</span><span> </span><span style="color:#608b4e;">"""A decorator to specify what columns should be retrieved."""
</span><span> cols = </span><span style="color:#d69d85;">", "</span><span>.join(columns)
</span><span> </span><span style="color:#569cd6;">def </span><span>outer(f):
</span><span> @wraps(f)
</span><span> </span><span style="color:#569cd6;">def </span><span>inner(</span><span style="color:#569cd6;">*</span><span>args, </span><span style="color:#569cd6;">**</span><span>kwargs):
</span><span> query = f(*args, </span><span style="color:#569cd6;">**</span><span>kwargs).split(</span><span style="color:#d69d85;">"*"</span><span>)
</span><span> start, end = query[</span><span style="color:#b5cea8;">0</span><span>].strip(), query[</span><span style="color:#b5cea8;">1</span><span>].strip()
</span><span> </span><span style="color:#569cd6;">return f</span><span style="color:#d69d85;">"</span><span>{start} {cols} {end}</span><span style="color:#d69d85;"> limit </span><span>{limit}</span><span style="color:#d69d85;">"
</span><span> </span><span style="color:#569cd6;">return </span><span>inner
</span><span> </span><span style="color:#569cd6;">return </span><span>outer
</span><span>
</span><span>
</span><span style="color:#569cd6;">def </span><span>as_csv(where=</span><span style="color:#d69d85;">"./result.csv"</span><span>):
</span><span> </span><span style="color:#608b4e;">"""A decorator to save the result as a csv file."""
</span><span> </span><span style="color:#569cd6;">def </span><span>outer(f):
</span><span> @wraps(f)
</span><span> </span><span style="color:#569cd6;">def </span><span>inner(</span><span style="color:#569cd6;">*</span><span>args, </span><span style="color:#569cd6;">**</span><span>kwargs):
</span><span> rset = f(*args, </span><span style="color:#569cd6;">**</span><span>kwargs)
</span><span> </span><span style="color:#569cd6;">with </span><span>open(where, </span><span style="color:#d69d85;">"w"</span><span>, newline=</span><span style="color:#d69d85;">""</span><span>) </span><span style="color:#569cd6;">as </span><span>dest:
</span><span> csv_writer = csv.writer(
</span><span> dest, delimiter=</span><span style="color:#d69d85;">","</span><span>, quoting=csv.QUOTE_ALL, lineterminator=</span><span style="color:#d69d85;">";</span><span style="color:#e3bbab;">\r\n</span><span style="color:#d69d85;">"
</span><span> )
</span><span> csv_writer.writerows(rset)
</span><span> </span><span style="color:#569cd6;">return </span><span>len(rset)
</span><span> </span><span style="color:#569cd6;">return </span><span>inner
</span><span> </span><span style="color:#569cd6;">return </span><span>outer
</span><span>
</span><span style="color:#608b4e;">###
</span><span>
</span><span>@as_csv()
</span><span>@connect(db)
</span><span>@columns(</span><span style="color:#d69d85;">"title"</span><span>, </span><span style="color:#d69d85;">"artistId"</span><span>)
</span><span style="color:#569cd6;">def </span><span>run(table=</span><span style="color:#d69d85;">"albums"</span><span>):
</span><span> query = </span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"select * from </span><span>{table}</span><span style="color:#d69d85;">"
</span><span> </span><span style="color:#569cd6;">return </span><span>query
</span><span>
</span><span style="color:#569cd6;">if </span><span>__name__ == </span><span style="color:#d69d85;">"__main__"</span><span>:
</span><span> print(run())
</span></code></pre>
<h1 id="8-even-more-interesting">8. Even more interesting:</h1>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>sqlite3
</span><span style="color:#9b9b9b;">import </span><span>pathlib
</span><span style="color:#9b9b9b;">from </span><span>functools </span><span style="color:#9b9b9b;">import </span><span>wraps
</span><span>
</span><span>here = pathlib.Path(__file__)
</span><span>db = here.parent.parent.parent / </span><span style="color:#d69d85;">"sqlite_stuff" </span><span>/ </span><span style="color:#d69d85;">"chinook" </span><span>/ </span><span style="color:#d69d85;">"chinook.db"
</span><span>
</span><span style="color:#569cd6;">def </span><span>connect(cls):
</span><span> conn = sqlite3.connect(db)
</span><span> conn.isolation_level = </span><span style="color:#569cd6;">None </span><span style="color:#608b4e;"># sqlite3.IMMEDIATE
</span><span> setattr(cls, </span><span style="color:#d69d85;">"conn"</span><span>, conn)
</span><span> </span><span style="color:#569cd6;">return </span><span>cls
</span><span>
</span><span>
</span><span style="color:#569cd6;">def </span><span>select(cls):
</span><span> </span><span style="color:#569cd6;">def </span><span>select(self):
</span><span> </span><span style="color:#569cd6;">return </span><span>cls.conn.execute(</span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"select * from </span><span>{cls.__name__}</span><span style="color:#d69d85;">"</span><span>).fetchall()
</span><span> setattr(cls, </span><span style="color:#d69d85;">"select"</span><span>, select)
</span><span> </span><span style="color:#569cd6;">return </span><span>cls
</span><span>
</span><span>
</span><span>@select
</span><span>@connect
</span><span style="color:#569cd6;">class </span><span>Albums:</span><span style="color:#569cd6;">...
</span><span>
</span><span style="color:#569cd6;">if </span><span>__name__ == </span><span style="color:#d69d85;">"__main__"</span><span>:
</span><span> a = Albums()
</span><span> print(a.select())
</span></code></pre>
<h1 id="m">8. Even more interesting, how decorators are used elsewhere:</h1>
<ul>
<li><a href="https://cubicweb.org/">Cubicweb</a> <a href="https://forge.extranet.logilab.fr/cubicweb/cubes/api/-/blob/branch/default/cubicweb_api/routes.py#L137">decorator</a> used to raise an AuthenticationError if no user is detected.</li>
<li><a href="https://flask.palletsprojects.com/en/2.2.x/">Flask</a> <a href="https://github.com/pallets/flask/blob/afc13b9390ae2e40f4731e815b49edc9ef52ed4b/examples/tutorial/flaskr/auth.py#L19">decorator</a> that redirects anonymous users to the login page.</li>
<li><a href="https://flask.palletsprojects.com/en/2.2.x/patterns/viewdecorators/#templating-decorator">Templating decorator</a> invented by the <a href="https://turbogears.org/">TurboGears</a> guys a while back to automatically render a template.</li>
<li><a href="https://www.djangoproject.com/">Django</a> <a href="https://github.com/django/django/blob/main/django/views/decorators/http.py">decorators</a> to manipulate <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers">http headers</a>.</li>
</ul>
<h1 id="9-more-on-this-topic">9. More on this topic:</h1>
<p>I hope that you found this post helpful. To learn about this topic, please, browse the following links:</p>
<ul>
<li><a href="https://peps.python.org/pep-0318/">PEP 318 – Decorators for Functions and Methods</a></li>
<li><a href="https://wiki.python.org/moin/PythonDecoratorLibrary">PythonDecoratorLibrary</a></li>
<li><a href="https://docs.python.org/3/library/functools.html">functools</a></li>
<li><a href="https://docs.python.org/3/glossary.html#term-decorator">decorator</a></li>
<li><a href="https://docs.python.org/3/library/sqlite3.html#">DB-API 2.0 interface for SQLite databases</a></li>
<li><a href="https://stackoverflow.com/questions/15357776/what-is-the-difference-between-functools-wraps-and-update-wrapper">The difference between wraps and update_wrapper</a></li>
<li><a href="https://www.artima.com/weblogs/viewpost.jsp?thread=241209">Python Decorators III: A Decorator-Based Build System</a></li>
<li><a href="https://xkcd.com/2692/">Interior decorating</a></li>
</ul>
Test driven interview2022-10-16T00:00:00+00:002022-10-16T00:00:00+00:00
Unknown
https://nskm.xyz/posts/tdi/<h1 id="introduction">Introduction:</h1>
<p>Test Driven Development <em>TDD</em> is a method of implementing software that rely on software requirements being converted to test cases before software is fully developed.</p>
<p>Red, Green and Refactor is the 3 phases of TDD. When followed, this order of steps helps ensure that you have tests for the code you are writing and you are writing only the code that you have to test for.</p>
<h1 id="context">Context:</h1>
<p>Let's suppose we should implement some requirements. Let's suppose all the requirements and specifications are written using unit tests. See this <a href="https://git.disroot.org/nsukami/interview">repository</a>. Our mission is to write enough code to make all the tests pass. Let's <a href="https://y.com.sb/watch?v=gptoH8rzG14">dive in</a>.</p>
<h1 id="toc">TOC:</h1>
<ul>
<li><a href="https://nskm.xyz/posts/tdi/#setup">Setup</a></li>
<li><a href="https://nskm.xyz/posts/tdi/#tests">Tests</a></li>
<li><a href="#">Requirements</a>
<ul>
<li><a href="https://nskm.xyz/posts/tdi/#firstreq">First requirement</a></li>
<li><a href="https://nskm.xyz/posts/tdi/#secondreq">Second requirement</a></li>
<li><a href="https://nskm.xyz/posts/tdi/#thirdreq">Third requirement</a></li>
<li><a href="https://nskm.xyz/posts/tdi/#fourthreq">Fourth requirement</a></li>
<li><a href="https://nskm.xyz/posts/tdi/#fifthreq">Fifth requirement</a></li>
<li><a href="https://nskm.xyz/posts/tdi/#lastreq">Last requirement</a></li>
</ul>
</li>
</ul>
<h2 id="setup">Setup</h2>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> git create-branch feat-1 </span><span style="color:#608b4e;"># create a new branch for feature work
</span><span>Basculement sur la nouvelle branche </span><span style="color:#d69d85;">'feat-1'
</span><span>(.env) ~/GIT/interview (feat-1)
</span><span>
</span><span>>> python3.10 -m venv .env </span><span style="color:#608b4e;"># create a virtual environment
</span><span>~/GIT/interview (feat-1)
</span><span>
</span><span>>> source .env/bin/activate </span><span style="color:#608b4e;"># activate virtual environment
</span><span>(.env) ~/GIT/interview (feat-1)
</span><span>
</span><span>>> pip install -r requirements.txt </span><span style="color:#608b4e;"># install requirements
</span><span>Collecting django<</span><span style="color:#b5cea8;">3</span><span>.2,>=3.1
</span><span> Using cached Django-3.1.14-py3-none-any.whl (7.8 MB)
</span><span>Collecting djangorestframework<</span><span style="color:#b5cea8;">3</span><span>.13,>=3.12
</span><span> Using cached djangorestframework-3.12.4-py3-none-any.whl (957 kB)
</span><span>Collecting sqlparse>=0.2.2
</span><span> Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
</span><span>Collecting asgiref<</span><span style="color:#b5cea8;">4</span><span>,>=3.2.10
</span><span> Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
</span><span>Collecting pytz
</span><span> Using cached pytz-2022.4-py2.py3-none-any.whl (500 kB)
</span><span>Installing collected packages: pytz, sqlparse, asgiref, django, djangorestframework
</span><span>Successfully installed asgiref-3.5.2 django-3.1.14 djangorestframework-3.12.4 pytz-2022.4 sqlparse-0.4.3
</span><span>WARNING: You are using pip version 22.0.4</span><span style="color:#569cd6;">; </span><span>however, version 22.3 is available.
</span><span>You should consider upgrading via the </span><span style="color:#d69d85;">'/home/nsukami/GIT/interview/.env/bin/python3.10 -m pip install --upgrade pip'</span><span> command.
</span><span>(.env) ~/GIT/interview (feat-1)
</span></code></pre>
<br>
<h2 id="tests">How to launch tests ?</h2>
<p>According to the <a href="https://docs.djangoproject.com/en/4.1/topics/testing/overview/#running-tests">documentation</a>, we can run tests using the test command of your project's manage.py utility. Let's <a href="https://eradman.com/entrproject/">run that command</a> when any of the Python files changes. We also tell Django to stop running the test suite after first failed test. In a different terminal, let's type one of the following commands:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> </span><span style="color:#608b4e;"># you can ask git ls-files command to watch for all the python files
</span><span>>> git ls-files</span><span style="color:#569cd6;"> -- </span><span style="color:#d69d85;">'*.py' ':!:*__.py' </span><span style="color:#569cd6;">| </span><span>entr sh -c </span><span style="color:#d69d85;">"python src/manage.py test --failfast --force-color -v 2 src"
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> </span><span style="color:#608b4e;"># you can ask find command to watch for all the python files
</span><span>>> find src -type f -name </span><span style="color:#d69d85;">"*.py"</span><span> -not -name </span><span style="color:#d69d85;">"__*" </span><span style="color:#569cd6;">| </span><span>entr sh -c </span><span style="color:#d69d85;">"python src/manage.py test --failfast --force-color -v 2 src"
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> </span><span style="color:#608b4e;"># the ack command seems more readable
</span><span>>> ack -f --python </span><span style="color:#569cd6;">| </span><span>entr sh -c </span><span style="color:#d69d85;">"python src/manage.py test --failfast --force-color -v 2 src"
</span></code></pre>
<br>
<p>Here is our first error:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>django.db.utils.OperationalError: no such table: inventory_product
</span></code></pre>
<br>
<h2 id="firstreq">1. First requirements: we need a Product model.</h2>
<p>We need to create a model named Product inside an application named inventory. <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L12">Somewhere</a> within the tests file, we can read:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span> </span><span style="color:#569cd6;">def </span><span>create_coke(self):
</span><span> </span><span style="color:#569cd6;">return </span><span>Product.objects.create(
</span><span> description=</span><span style="color:#d69d85;">'Coca-Cola'</span><span>,
</span><span> unit_price=</span><span style="color:#b5cea8;">500</span><span>,
</span><span> stock=</span><span style="color:#b5cea8;">10
</span><span> )
</span></code></pre>
<br>
<p>Let's <a href="https://git.disroot.org/nsukami/interview/src/branch/feat-1/src/modules/inventory/models.py#L4">open</a> the <code>models.py</code> file and write the following:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">from </span><span>django.db </span><span style="color:#9b9b9b;">import </span><span>models
</span><span>
</span><span style="color:#569cd6;">class </span><span>Product(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> description = models.CharField(max_length=</span><span style="color:#b5cea8;">200</span><span>)
</span><span> unit_price = models.FloatField()
</span><span> stock = models.IntegerField()
</span></code></pre>
<br>
<p>After running the migration scripts:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> python src/manage.py makemigrations </span><span style="color:#569cd6;">&& </span><span>python src/manage.py migrate
</span></code></pre>
<br>
<p>The tests tells us to return an <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/201">http status 201</a> after Product creation. Right now, we have a 404. We need to:</p>
<ul>
<li>create a REST API.</li>
<li>we should be able to create a Product via the API.</li>
<li>and the returned http status should be 201 _created.</li>
</ul>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>=</span><span style="background-color:#282828;color:#d69d85;">=====================================================================</span><span>
</span><span>FAIL: test_create_product (modules.inventory.tests.ProductTestCase)
</span><span>----------------------------------------------------------------------
</span><span>Traceback (most recent call last):
</span><span> File </span><span style="color:#d69d85;">"/home/nsukami/GIT/interview/src/modules/inventory/tests.py"</span><span>, line 45, in test_create_product
</span><span> self.assertEqual(response.status_code, 201)
</span><span>AssertionError: 404 != 201
</span><span>
</span><span>----------------------------------------------------------------------
</span><span>Ran 1 test in 0.010s
</span><span>
</span><span>FAILED (failures=1)
</span></code></pre>
<br>
<h2 id="secondreq">We need a REST API for managing a Product.</h2>
<p>Another <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L44">part</a> of the tests is telling us we need an url named <code>api/products/</code>. Let's create a <a href="https://www.django-rest-framework.org/api-guide/serializers/#modelserializer">serializer</a>, a <a href="https://www.django-rest-framework.org/api-guide/viewsets/#modelviewset">viewset</a> and a <a href="https://www.django-rest-framework.org/api-guide/routers/">router</a>. Here is the code:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># /src/modules/inventory/views.py
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>serializers </span><span style="color:#608b4e;"># type:ignore
</span><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>viewsets
</span><span style="color:#9b9b9b;">from .</span><span>models </span><span style="color:#9b9b9b;">import </span><span>Product
</span><span>
</span><span style="color:#569cd6;">class </span><span>ProductSerializer(</span><span style="color:#4ec9b0;">serializers.ModelSerializer</span><span>):
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> model = Product
</span><span> fields = </span><span style="color:#d69d85;">"__all__"
</span><span>
</span><span style="color:#569cd6;">class </span><span>ProductsViewSet(</span><span style="color:#4ec9b0;">viewsets.ModelViewSet</span><span>):
</span><span> queryset = Product.objects.all()
</span><span> serializer_class = ProductSerializer
</span><span>
</span><span style="color:#608b4e;"># /src/pos/urls.py
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>django.urls </span><span style="color:#9b9b9b;">import </span><span>path, include
</span><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>routers
</span><span style="color:#9b9b9b;">from </span><span>modules.inventory.views </span><span style="color:#9b9b9b;">import </span><span>ProductsViewSet
</span><span>
</span><span>router = routers.DefaultRouter()
</span><span>router.register(</span><span style="color:#569cd6;">r</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">products</span><span style="color:#d69d85;">"</span><span>, ProductsViewSet)
</span><span>
</span><span>urlpatterns = [
</span><span> path(</span><span style="color:#d69d85;">"api/"</span><span>, include(router.urls)),
</span><span>]
</span><span>
</span></code></pre>
<br>
<p>After the changes, we notice some progress. The test tells us that if we attempt to delete a Product, we should receive an <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/405">http status 405</a> instead of 204. Litteraly, we have been able to delete a Product while it shouldn't be allowed:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>=</span><span style="background-color:#282828;color:#d69d85;">=====================================================================</span><span>
</span><span>FAIL: test_delete_product (modules.inventory.tests.ProductTestCase)
</span><span>----------------------------------------------------------------------
</span><span>Traceback (most recent call last):
</span><span> File </span><span style="color:#d69d85;">"/home/nsukami/GIT/interview/src/modules/inventory/tests.py"</span><span>, line 142, in test_delete_product
</span><span> self.assertEqual(response.status_code, 405)
</span><span>AssertionError: 204 != 405
</span><span>
</span><span>----------------------------------------------------------------------
</span><span>Ran 2 tests in 0.011s
</span><span>
</span><span>FAILED (failures=1)
</span></code></pre>
<br>
<h2 id="thirdreq">A Product cannot be deleted.</h2>
<p>I should have take the time to carefully read all the tests. <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L139">Indeed</a>, one of them clearly states that we can't delete a Product. Obviously, the ModelViewSet is not a good choice. We should update our viewset in a way that forbids product deletion. The operations: <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L36">creating</a>, <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L18">listing</a>, <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L95">updating</a>, <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py#L57">getting one product</a> should still be permitted. To achieve that, we'll use some <a href="https://www.django-rest-framework.org/api-guide/generic-views/#mixins">mixins</a> offered by the framework. The product viewset becomes:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>viewsets, mixins
</span><span>
</span><span style="color:#569cd6;">class </span><span>ProductsViewSet(
</span><span> </span><span style="color:#4ec9b0;">mixins.CreateModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">mixins.ListModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">mixins.UpdateModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">mixins.RetrieveModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">viewsets.GenericViewSet</span><span>,
</span><span>):
</span><span> queryset = Product.objects.all()
</span><span> serializer_class = ProductSerializer
</span></code></pre>
<br>
<h3 id="let-s-commit-what-have-been-done-so-far">Let's commit what have been done so far.</h3>
<p>All the tests related to Product are passing. We can commit and push what we've done so far.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> git commit -m </span><span style="color:#d69d85;">"Add Product API"
</span><span>[feat-1 a11ffeb] Add Product API
</span><span> 5 files changed, 76 insertions(+), 6 deletions(-)
</span><span> create mode 100644 src/modules/inventory/migrations/0001_initial.py
</span><span> rewrite src/modules/inventory/views.py (98%)
</span><span> create mode 100644 src/modules/orders/migrations/0001_initial.py
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> git push origin feat-1
</span><span>Énumération des objets: 25, fait.
</span><span>Décompte des objets: 100% (24/24), fait.
</span><span>Compression par delta en utilisant jusqu</span><span style="color:#d69d85;">'à 4 fils d'</span><span>exécution
</span><span>Compression des objets: 100% (14/14), fait.
</span><span>Écriture des objets: 100% (14/14), 1.80 Kio </span><span style="color:#569cd6;">| </span><span>1.80 Mio/s, fait.
</span><span>Total 14 (delta 6), réutilisés 0 (delta 0), réutilisés du pack 0
</span><span>remote:
</span><span>remote: Create a new pull request for </span><span style="color:#d69d85;">'feat-1'</span><span>:
</span><span>remote: https://git.disroot.org/nsukami/interview/compare/master...feat-1
</span><span>remote:
</span><span>remote: . Processing 1 references
</span><span>remote: Processed 1 references in total
</span><span>To https://git.disroot.org/nsukami/interview.git
</span><span> * </span><span style="color:#569cd6;">[</span><span>new branch</span><span style="color:#569cd6;">]</span><span> feat-1 -> feat-1
</span></code></pre>
<img style="display: block; margin: 0 auto; width:100; height:100" src="/images/ofcourse.gif" alt="Of course" title="Of course"/>
<h2 id="fourthreq">We need another model named Order.</h2>
<p>All the tests located in the <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/inventory/tests.py">inventory</a> application are passing. This is why we decided to commit and push. There is one test currently failing, it it located inside the <a href="https://git.disroot.org/nsukami/interview/src/branch/master/src/modules/orders/tests.py">orders</a> application. The test tells us we need to create a model named Order:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>=</span><span style="background-color:#282828;color:#d69d85;">=====================================================================</span><span>
</span><span>FAIL: test_create_order (modules.orders.tests.OrdersTestCase)
</span><span>----------------------------------------------------------------------
</span><span>Traceback (most recent call last):
</span><span> File </span><span style="color:#d69d85;">"/home/nsukami/GIT/interview/src/modules/orders/tests.py"</span><span>, line 87, in test_create_order
</span><span> self.assertEqual(response.status_code, 201)
</span><span>AssertionError: 404 != 201
</span><span>
</span><span>----------------------------------------------------------------------
</span><span>Ran 7 tests in 0.054s
</span><span>
</span><span>FAILED (failures=1)
</span></code></pre>
<br>
<p>Let's continue reading the tests. <a href="https://git.disroot.org/nsukami/interview/src/branch/feat-1/src/modules/orders/tests.py#L90">Line 90</a>, we read that an item should have 4 attributes: <code>description, quantity, unit_price, total</code>. We also read that an Order has 2 attributes: <code>items</code> and <code>total</code>:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>order = Order.objects.get()
</span><span>expected = {
</span><span> </span><span style="color:#d69d85;">'id'</span><span>: order.id,
</span><span> </span><span style="color:#d69d85;">'items'</span><span>: [
</span><span> {
</span><span> </span><span style="color:#d69d85;">'description'</span><span>: </span><span style="color:#d69d85;">'Coca-Cola'</span><span>,
</span><span> </span><span style="color:#d69d85;">'quantity'</span><span>: </span><span style="color:#b5cea8;">1</span><span>,
</span><span> </span><span style="color:#d69d85;">'unit_price'</span><span>: </span><span style="color:#b5cea8;">500</span><span>,
</span><span> </span><span style="color:#d69d85;">'total'</span><span>: </span><span style="color:#b5cea8;">500</span><span>,
</span><span> },
</span><span> {
</span><span> </span><span style="color:#d69d85;">'description'</span><span>: </span><span style="color:#d69d85;">'Potato Chips'</span><span>,
</span><span> </span><span style="color:#d69d85;">'quantity'</span><span>: </span><span style="color:#b5cea8;">2</span><span>,
</span><span> </span><span style="color:#d69d85;">'unit_price'</span><span>: </span><span style="color:#b5cea8;">1000</span><span>,
</span><span> </span><span style="color:#d69d85;">'total'</span><span>: </span><span style="color:#b5cea8;">2000</span><span>,
</span><span> }
</span><span> ],
</span><span> </span><span style="color:#d69d85;">'total'</span><span>: </span><span style="color:#b5cea8;">2500</span><span>,
</span><span>}
</span></code></pre>
<p>So, our models may look like the following:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">class </span><span>Order(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> @property
</span><span> </span><span style="color:#569cd6;">def </span><span>total(self):
</span><span> </span><span style="color:#608b4e;"># this field is just a sum, depending on all the items in the current Order
</span><span> </span><span style="color:#569cd6;">return </span><span>sum([x.total </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>self.items.all()])
</span><span>
</span><span style="color:#569cd6;">class </span><span>OrderItem(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> order = models.ForeignKey(Order, related_name=</span><span style="color:#d69d85;">"items"</span><span>, on_delete=models.CASCADE)
</span><span> </span><span style="color:#608b4e;"># For being able to retrieve the description and the unit_price fields ?
</span><span> </span><span style="color:#608b4e;"># Should I add Product as a foreign key ?
</span><span> </span><span style="color:#608b4e;"># Should I duplicate the fields ?
</span><span>
</span><span> @property
</span><span> </span><span style="color:#569cd6;">def </span><span>total(self):
</span><span> </span><span style="color:#608b4e;"># the total is obviouly a property that will be calculated
</span><span> </span><span style="color:#608b4e;"># based on the number of items in the Order & the unit price for each item
</span><span> </span><span style="color:#569cd6;">return </span><span>float(self.unit_price * self.quantity)
</span></code></pre>
<br>
<h2 id="fifthreq">We should be able to update the stock depending on the number of items ordered:</h2>
<p>The <a href="https://git.disroot.org/nsukami/interview/src/branch/feat-1/src/modules/orders/tests.py#L114">Line 114</a> of the tests tells us, a Product should have a method to refresh its stock:</p>
<pre data-lang="pyhon" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-pyhon "><code class="language-pyhon" data-lang="pyhon"><span>self.chips.refresh_from_db()
</span><span>self.assertEqual(self.chips.stock, 8)
</span></code></pre>
<p>Let's update our Product model and add the new method. This method will substract the number of Product ordered from the current stock:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">from </span><span>django.db </span><span style="color:#9b9b9b;">import </span><span>models
</span><span>
</span><span style="color:#569cd6;">class </span><span>Product(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> description = models.CharField(max_length=</span><span style="color:#b5cea8;">200</span><span>)
</span><span> unit_price = models.FloatField()
</span><span> stock = models.IntegerField()
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>refresh_from_db(self):
</span><span> </span><span style="color:#9b9b9b;">from </span><span>modules.orders.models </span><span style="color:#9b9b9b;">import </span><span>OrderItem
</span><span>
</span><span> self.stock = self.stock - sum(
</span><span> oi.quantity </span><span style="color:#569cd6;">for </span><span>oi </span><span style="color:#569cd6;">in </span><span>OrderItem.objects.filter(id=self.id)
</span><span> )
</span><span> self.save()
</span></code></pre>
<br>
<h2 id="lastreq">We should be able to update a Product without modifiying already made Orders:</h2>
<p>After reading the <a href="https://git.disroot.org/nsukami/interview/src/branch/feat-1/src/modules/orders/tests.py#L141">Line 141</a>, we know that the fields <code>description and unit_price</code> will be duplicated inside the OrderItem model. During the OrderItem creation, we should copy those fields from the Product model. This task will be achieved using our serializers:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">class </span><span>ItemSerializer(</span><span style="color:#4ec9b0;">serializers.ModelSerializer</span><span>):
</span><span> quantity = serializers.IntegerField(required=</span><span style="color:#569cd6;">False</span><span>)
</span><span> unit_price = serializers.FloatField(required=</span><span style="color:#569cd6;">False</span><span>)
</span><span> description = serializers.CharField(required=</span><span style="color:#569cd6;">False</span><span>)
</span><span>
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> model = OrderItem
</span><span> fields = [</span><span style="color:#d69d85;">"description"</span><span>, </span><span style="color:#d69d85;">"quantity"</span><span>, </span><span style="color:#d69d85;">"unit_price"</span><span>, </span><span style="color:#d69d85;">"total"</span><span>]
</span><span>
</span><span>
</span><span style="color:#569cd6;">class </span><span>OrderSerializer(</span><span style="color:#4ec9b0;">serializers.ModelSerializer</span><span>):
</span><span> items = ItemSerializer(many=</span><span style="color:#569cd6;">True</span><span>)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>create(self, validated_data):
</span><span> order = Order.objects.create()
</span><span> </span><span style="color:#608b4e;"># I want to be able to access the id, the id represent the Product id
</span><span> </span><span style="color:#608b4e;"># So, I'm using initial_data instead of validated_data
</span><span> </span><span style="color:#608b4e;"># I'll need to find a proper way
</span><span> </span><span style="color:#569cd6;">for </span><span>item_data </span><span style="color:#569cd6;">in </span><span>self.initial_data[</span><span style="color:#d69d85;">"items"</span><span>]:
</span><span> product = Product.objects.get(id=item_data[</span><span style="color:#d69d85;">"id"</span><span>])
</span><span> OrderItem.objects.create(
</span><span> id=product.id,
</span><span> order=order,
</span><span> description=product.description,
</span><span> unit_price=product.unit_price,
</span><span> quantity=item_data[</span><span style="color:#d69d85;">"quantity"</span><span>],
</span><span> )
</span><span> </span><span style="color:#569cd6;">return </span><span>order
</span><span>
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> model = Order
</span><span> fields = [</span><span style="color:#d69d85;">"id"</span><span>, </span><span style="color:#d69d85;">"items"</span><span>, </span><span style="color:#d69d85;">"total"</span><span>]
</span><span>
</span><span>
</span><span style="color:#569cd6;">class </span><span>OrdersViewSet(
</span><span> </span><span style="color:#4ec9b0;">mixins.CreateModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">mixins.ListModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">mixins.RetrieveModelMixin</span><span>,
</span><span> </span><span style="color:#4ec9b0;">viewsets.GenericViewSet</span><span>,
</span><span>):
</span><span>
</span><span> queryset = Order.objects.all()
</span><span> serializer_class = OrderSerializer
</span></code></pre>
<br>
<h1 id="conclusion">Conclusion:</h1>
<p>It's been a long time since I've written Django and DRF code. I really enjoyed the exercise. The final result can be found <a href="https://git.disroot.org/nsukami/interview/src/branch/feat-1/">here</a>.</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> python src/manage.py test src --failfast
</span><span>Creating test database for alias </span><span style="color:#d69d85;">'default'</span><span>...
</span><span>System check identified no issues (0 silenced).
</span><span>............
</span><span>----------------------------------------------------------------------
</span><span>Ran 12 tests in 0.093s
</span><span>
</span><span>OK
</span><span>Destroying test database for alias </span><span style="color:#d69d85;">'default'</span><span>...
</span></code></pre>
<br>
<h1 id="while-i-still-have-your-attention">While I still have your attention:</h1>
<p>A <a href="https://djangochat.com/episodes/django-unicorn-adam-hill">conversation</a> with Adam Hill about writing software, the perfect settings file set up, and more. Adam is a software engineer at The Motley Fool and the author of multiple open source packages including <a href="https://www.django-unicorn.com/">Django Unicorn</a>.
<br></p>
<p>The videos for the <a href="https://www.youtube.com/watch?v=0kOLsFQGxyk&list=PLY_che_OEsX3eIKKLUKvQXfVJhCHYWATM">DjangoCon Europe 2022</a> are available. Can someone tell me please when the <a href="https://www.djangoproject.com/weblog/2019/nov/18/introducing-djangocon-africa/">DjangoCon Africa</a> event will be held ? So I can prepare myself ?</p>
<br>
<h1 id="more-on-the-topic">More on the topic:</h1>
<p>I really hope you have learned something. To dive even more deeper in the topic, let me please recommend the following links:</p>
<ul>
<li><a href="https://xkcd.com/125/">Marketing interview</a></li>
<li><a href="https://en.wikipedia.org/wiki/Test-driven_development">Test Driven Development</a></li>
<li><a href="https://docs.python.org/3/library/unittest.html">Unit testing framework</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/Testing">Testing a Django web application</a></li>
<li><a href="https://test-driven-django-development.readthedocs.io/en/latest/">Test Driven Web Development with Django</a></li>
<li><a href="https://docs.djangoproject.com/en/4.1/topics/testing/overview/">Writing and running tests</a></li>
<li><a href="https://google.aip.dev/">API Improvement Proposal</a></li>
<li><a href="https://nskm.xyz/posts/dsfpa/">Django single file project</a></li>
<li><a href="https://yewtu.be/watch?v=rR4n-0KYeKQ">How we write/review code in big tech companies</a></li>
</ul>
Linear regression2022-07-24T00:00:00+00:002022-07-24T00:00:00+00:00
Unknown
https://nskm.xyz/posts/linreg/<p>For the <a href="https://dit.sn">Dakar Institute of Technology</a>, I give Python introductory classes. During one of my sessions I was asked about simple linear regression with Python. Let the music <a href="https://www.youtube.com/watch?v=adBfKzfemtI">play</a></p>
<center>
<img src="https://nskm.xyz/processed_images/disclaimer_not_ds.0eb9147b61b70fa9.jpg" />
</center>
<br>
<br>
<h1 id="toc">TOC:</h1>
<ul>
<li><a href="https://nskm.xyz/posts/linreg/#introduction">Introduction</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#first-solution">First solution</a>
<ul>
<li><a href="https://nskm.xyz/posts/linreg/#reading-the-data-set">Reading the data set</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#plotting-the-data-set">Plotting the data set</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#splitting-the-data-set">Splitting the data set</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#ordinary-least-squares">Least squares estimates</a>
<ul>
<li><a href="https://nskm.xyz/posts/linreg/#mean">Mean</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#variance">Variance</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#covariance">Covariance</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#estimates">Estimates</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#prediction">Prediction</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#error">How good is the model</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#final-plot">Regression line</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="https://nskm.xyz/posts/linreg/#class">The complete program</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#statistics">Another solution using the statistics module</a></li>
<li><a href="https://nskm.xyz/posts/linreg/#code">Just give me the code please</a></li>
</ul>
<h1 id="introduction">Introduction:</h1>
<p>Simple linear regression is a commonly used type of predictive analysis. The idea of regression is to examine if a set of predictor variables do a good job in predicting an outcome variable.</p>
<p>Linear regression models are used in many industries such as healthcare (predicting disease from biological factors), economics (predicting growth), businesses (predicting product sales), etc ...</p>
<p>For simple linear regression, the relationship between the independent variable <strong>x</strong> and the dependent variable <strong>y</strong> is expressed as: <code>y = slope * x + intercept + noise</code>, where:</p>
<ul>
<li><em>slope</em> and <em>intercept</em> are the regression parameters that are estimated</li>
<li><em>noise</em> is the variability of the data (difference between predicted and actual values of <strong>y</strong>).</li>
</ul>
<br>
<h1 id="first-solution">First solution:</h1>
<p>Let's write a Python program to achieve this task. We'll use the following <a href="/assets/dataset.txt">data set</a></p>
<h2 id="reading-the-data-set">1. Reading the data set:</h2>
<p>To read the data, we'll write a function in charge of opening and reading the file containing our data. We assume each line will consist of an int value, followed by a tabulation and a float value. This function will:</p>
<ul>
<li>open the file</li>
<li>get all the lines except the first one (the header)</li>
<li>convert first column to int, convert second column to float</li>
<li>return a tuple containing 2 things:</li>
</ul>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>read_text(
</span><span> filename: t.Union[pathlib.Path, t.AnyStr],
</span><span>) -> t.Tuple[t.Collection[float], t.Collection[float]]:
</span><span> </span><span style="color:#569cd6;">with </span><span>open(filename, </span><span style="color:#d69d85;">"r"</span><span>) </span><span style="color:#569cd6;">as </span><span>f:
</span><span> slines = [line.strip().split(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\t</span><span style="color:#d69d85;">"</span><span>) </span><span style="color:#569cd6;">for </span><span>line </span><span style="color:#569cd6;">in </span><span>[li </span><span style="color:#569cd6;">for </span><span>li </span><span style="color:#569cd6;">in </span><span>f.readlines()[</span><span style="color:#b5cea8;">1</span><span>:]]]
</span><span> nlines = [(int(x), float(y)) </span><span style="color:#569cd6;">for </span><span>x, y </span><span style="color:#569cd6;">in </span><span>slines]
</span><span> X, Y = zip(*nlines)
</span><span> </span><span style="color:#569cd6;">return </span><span>X, Y
</span></code></pre>
<br>
<p>Bonjour</p>
<h2 id="plotting-the-data-set">2. Plotting the data set:</h2>
<p>We'll have to do some plotting. The following program is just one we use to setup our <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.subplots.html#matplotlib-pyplot-subplots">plots</a>:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>ax: mpl.axes.Axes
</span><span>_, ax = plt.subplots()
</span><span>ax.set_xlabel(</span><span style="color:#d69d85;">"X-coordinates"</span><span>)
</span><span>ax.set_ylabel(</span><span style="color:#d69d85;">"Y-coordinates "</span><span>)
</span><span>ax.set_title(</span><span style="color:#d69d85;">"Graphical representation of our points"</span><span>)
</span><span>ax.grid(</span><span style="color:#569cd6;">True</span><span>)
</span></code></pre>
<br>
<p>The first step in finding a linear regression equation is to determine if there is a relationship between the two variables. To determine if there is a relationship between x and y we can scatter plot the data set.</p>
<p>Scatter plots are used to observe relationships between variables. We'll need one function to do a scatter plot: put dots on our graph and see each individual data point.</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>scatter_plot(
</span><span> x: t.Collection[float], y: t.Collection[float], ax: mpl.axes.Axes
</span><span>) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> ax.scatter(x, y, color=</span><span style="color:#d69d85;">"red"</span><span>)
</span></code></pre>
<p>Using this function with our <a href="/assets/dataset.txt">data set</a>, we can see the following chart:</p>
<center>
<img src="https://nskm.xyz/processed_images/scatter_plot.04b695a96b3966b3.png" />
</center>
<br>
<br>
<p>We'll need a function to draw a line. If we correctly find estimates for our regression parameters (slope and the intercept), we should be able to draw a line that will go through the cloud of data points:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>line_plot(a: int, b: int, ax: mpl.axes.Axes) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> x = [</span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#b5cea8;">130</span><span>]
</span><span> y = [a * x + b </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>x]
</span><span> ax.plot(x, y)
</span></code></pre>
<br>
<h2 id="splitting-the-data-set">3. Splitting the data set:</h2>
<p>We don’t want our model to over learn from the data and perform poorly when presented data never seen before. We also need to have a mechanism to assess how well our model is generalizing. Hence, we'll need to separate our input data into <em>at least</em> 2 sets: training and testing:</p>
<h4 id="1-training-data-set">1. Training data set:</h4>
<p>This is the set of data that will be used only for learning. We will use only one part of the data set to find the best parameters for our model.</p>
<h4 id="2-testing-data-set">2. Testing data set:</h4>
<p>It would be best if we evaluate the model with new data that hasn’t been seen by the model before. So we need a set of data used to provide an evaluation without bias of the model that was fitted on the training data set.</p>
<p>Let's write a function that will split our data. We'll keep 80% for the training and 20% for testing. This function will:</p>
<ul>
<li>randomly pick 20% of all the indexes in our collection</li>
<li>with those 20% indexes, create a list containing 20% of the collections</li>
<li>from those 20% indexes, create a list containing 80% of the collections</li>
<li>return all the sets</li>
</ul>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>split_seq = t.Tuple[
</span><span> t.Collection[float], t.Collection[float], t.Collection[float], t.Collection[float]
</span><span>]
</span><span>
</span><span style="color:#569cd6;">def </span><span>split_train_test(
</span><span> X: t.Collection[float], Y: t.Collection[float], frac: float = </span><span style="color:#b5cea8;">0.2
</span><span>) -> split_seq:
</span><span>
</span><span> indexes = list(range(len(X)))
</span><span> indexes_test = set(random.sample(indexes, int(frac * len(X))))
</span><span>
</span><span> </span><span style="color:#608b4e;"># get 20% of Xs based on the indexes, do the same for Ys
</span><span> X_test = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(X) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">in </span><span>indexes_test]
</span><span> Y_test = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(Y) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">in </span><span>indexes_test]
</span><span>
</span><span> </span><span style="color:#608b4e;"># get 80% of Xs based on the indexes, do the same for Ys
</span><span> X_train = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(X) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">not in </span><span>indexes_test]
</span><span> Y_train = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(Y) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">not in </span><span>indexes_test]
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span>X_train, X_test, Y_train, Y_test
</span></code></pre>
<p>Let's visualize how the 2 sets are rendered in a scatter plot:</p>
<center>
<img src="https://nskm.xyz/processed_images/train_test_split.01769c608aeb949a.png" />
</center>
<br>
<br>
<h1 id="ordinary-least-squares">4. Least squares for the slope and the intercept:</h1>
<p>How do we estimate our parameters α and β ? That’s the learning procedure. There are different optimization techniques to implement this part. To determine the optimum values of these two parameters, we should find the line that best fit the data set.</p>
<p>To determine the optimum values of these two parameters, we need to find the <strong>slope</strong> and the <strong>intercept</strong> so that when we sum all the errors between the predicted y values and the observed y values, this sum should be the least possible.</p>
<p>The following chart will show us all the distances (or all the differences, or all the errors) between each point on the dataset and the regression line:</p>
<center>
<img src="https://nskm.xyz/processed_images/plotting_OLS.9924776937a70e5c.png" />
</center>
<br>
<br>
<p>Let's see how the chart looks like when the parameters are not correctly estimated <em>(the regression line does not "fit" the data set)</em> :</p>
<center>
<img src="https://nskm.xyz/processed_images/plotting_bad_OLS.3f873a7960094d1f.png" />
</center>
<br>
<br>
<p>And here is the function used to plot the errors. This function will:</p>
<ul>
<li>Take the estimated coefficients α and β</li>
<li>Take the list of X and the list of Y</li>
<li>For all points in our data set:
<ul>
<li>predict <code>y</code> based on α and β</li>
<li>draw a dashed line between 2 points: <code>(x, y)</code> and <code>(x, y_pred)</code></li>
</ul>
</li>
</ul>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>plot_errors(
</span><span> a: float,
</span><span> b: float,
</span><span> X: t.Collection[float],
</span><span> Y: t.Collection[float],
</span><span> ax: mpl.axes.Axes,
</span><span>) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> </span><span style="color:#569cd6;">for </span><span>x, y </span><span style="color:#569cd6;">in </span><span>zip(X, Y):
</span><span> y_pred = a * x + b
</span><span> ax.plot([x, x], [y, y_pred], </span><span style="color:#d69d85;">"bo"</span><span>, label=</span><span style="color:#d69d85;">"Errors"</span><span>, linestyle=</span><span style="color:#d69d85;">"dashed"</span><span>)
</span></code></pre>
<br>
<h1 id="the-total-error-of-this-model-is-the-sum-of-all-the-errors-for-each-point">The total error of this model is the sum of all the errors for each point:</h1>
<p>We should sum all the errors and look for the least error. Here is the formula:</p>
<p><em>The SSE (Sum of Squared Errors) is equal to the sum of the square of Y observed - Y calculated, for all point in the data set.</em></p>
<center>
<img src="https://nskm.xyz/processed_images/lse_formula.1b56ea6087956cd1.png" />
</center>
<br>
<br>
<p>If we do all the correct <a href="/assets/Simple_Linear_Regression_Leastsquares.pdf">algebraic manipulation</a>, we should find that:</p>
<center>
<img src="https://nskm.xyz/processed_images/slope_intercept_formulas.4b449a01bfa75294.png" />
</center>
<br>
<p>Seeing those formulas, those familiar with statistics already know we'll need to define a function to calculate a mean, another function to calculate a variance and a function to calculate a covariance. βeta hat can be defined as Covariance(X, Y) / Variance(X).</p>
<h2 id="mean">5.1 Mean:</h2>
<p>The <em>arithmetic</em> mean, is a measure of central tendency of a finite set of numbers: specifically, the sum of the values divided by the number of values:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>mean(lst: t.Collection[float]) -> float:
</span><span> </span><span style="color:#569cd6;">return </span><span>sum(lst) / len(lst)
</span></code></pre>
<br>
<h2 id="variance">5.2 Variance:</h2>
<p>The variance is the measure of the <strong>spread</strong> in a data set. The variance is a measure of how far each number in the data set is from the mean, and thus from every other number in the data set. The higher the variance, the more spread out the variables are.</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>variance(lst: t.Collection[float]) -> float:
</span><span> m = mean(lst)
</span><span> </span><span style="color:#569cd6;">return </span><span>sum([(x - m) ** </span><span style="color:#b5cea8;">2 </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>lst]) / len(lst)
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>variance: 582.8381391772394
</span></code></pre>
<p>There is 2 ways to plot the variance. We can do it manually, or we can use a <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.boxplot.html#matplotlib-pyplot-boxplot">box plot</a>:</p>
<center>
<img src="https://nskm.xyz/processed_images/plotting_variance.b71e17d987149681.png" />
</center>
<img style="display: block; margin: 0 auto" src="/assets/jcvd_split.gif" alt="Jean-Claude Van Damme split looks like a box plot" title="Jean-Claude Van Damme split looks like a box plot"/>
<br>
<p>Here is the function used to plot the variance:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>plot_variance(
</span><span> X: t.Collection[float],
</span><span> Y: t.Collection[float],
</span><span>) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> fig: mpl.figure.Figure
</span><span> ax: mpl.axes.Axes
</span><span>
</span><span> _, ax = plt.subplots(</span><span style="color:#b5cea8;">2</span><span>)
</span><span> ax[</span><span style="color:#b5cea8;">1</span><span>].set_xlabel(</span><span style="color:#d69d85;">"X-coordinates "</span><span>)
</span><span> ax[</span><span style="color:#b5cea8;">1</span><span>].set_ylabel(</span><span style="color:#d69d85;">"Y-coordinates"</span><span>)
</span><span> ax[</span><span style="color:#b5cea8;">0</span><span>].set_title(</span><span style="color:#d69d85;">"Graphical representation of variance"</span><span>)
</span><span> ax[</span><span style="color:#b5cea8;">1</span><span>].grid(</span><span style="color:#569cd6;">True</span><span>)
</span><span>
</span><span> </span><span style="color:#608b4e;"># box plot
</span><span> ax[</span><span style="color:#b5cea8;">0</span><span>].boxplot([X], vert=</span><span style="color:#569cd6;">False</span><span>, showmeans=</span><span style="color:#569cd6;">True</span><span>)
</span><span>
</span><span> </span><span style="color:#608b4e;"># mean line
</span><span> ax[</span><span style="color:#b5cea8;">1</span><span>].axvline(x=mean(X), ymin=</span><span style="color:#b5cea8;">0</span><span>, ymax=</span><span style="color:#b5cea8;">130</span><span>, color=</span><span style="color:#d69d85;">"orange"</span><span>)
</span><span>
</span><span> </span><span style="color:#608b4e;"># distance from point to mean line
</span><span> x_mean = mean(X)
</span><span> </span><span style="color:#569cd6;">for </span><span>x, y </span><span style="color:#569cd6;">in </span><span>zip(X, Y):
</span><span> ax[</span><span style="color:#b5cea8;">1</span><span>].plot(
</span><span> [x, x_mean],
</span><span> [y, y],
</span><span> marker=</span><span style="color:#d69d85;">"o"</span><span>,
</span><span> color=</span><span style="color:#d69d85;">"green"</span><span>,
</span><span> label=</span><span style="color:#d69d85;">"Mean"</span><span>,
</span><span> linestyle=</span><span style="color:#d69d85;">"dotted"</span><span>,
</span><span> )
</span><span> plt.show()
</span></code></pre>
<br>
<h2 id="covariance">5.3 Covariance:</h2>
<p>The covariance is a measure of the relationship between two variables. Covariance is the extent to which the variance in one variable depends on another variable. The higher the covariance, the stronger the relationship.</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>covariance(X: t.Sequence[float], Y: t.Sequence[float]) -> float:
</span><span> mean_x, mean_y, length_x = mean(X), mean(Y), len(X)
</span><span> covar = </span><span style="color:#b5cea8;">0.0
</span><span> </span><span style="color:#569cd6;">for </span><span>i </span><span style="color:#569cd6;">in </span><span>range(len(X)):
</span><span> covar += (X[i] - mean_x) * (Y[i] - mean_y)
</span><span> </span><span style="color:#569cd6;">return </span><span>covar / length_x
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>covariance: 2059.003344867359
</span></code></pre>
<br>
<h2 id="estimates">5.4 Estimates:</h2>
<p>Now that we have a function for calculating the variance and another one for calculating the covariance, we can write a function that will calculate our slope and our intercept using this 2 formulas seen earlier:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>estimate_coefficients(X: t.Collection[float], Y: t.Collection[float]) -> t.Tuple[float, float]:
</span><span> b1 = covariance(X, Y) / variance(X)
</span><span> b0 = mean(Y) - b1 * mean(X)
</span><span> </span><span style="color:#569cd6;">return </span><span>b0, b1
</span></code></pre>
<br>
<h2 id="prediction">5.5. Prediction:</h2>
<p>Now that we have extracted our coefficients, let's see if we can make a prediction, we'll use the testing data set:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>b, a = estimate_coefficients(X_train, Y_train)
</span><span>Y_predicted = [a * x + b </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>X_test]
</span><span style="color:#569cd6;">for </span><span>i, j </span><span style="color:#569cd6;">in </span><span>zip(Y_test, Y_predicted):
</span><span> print(i, </span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\t</span><span style="color:#d69d85;">'</span><span>, j) </span><span style="color:#608b4e;"># print Y observed in testing set next to Y predicted using our coefficients
</span><span>
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>> python3 script.py
</span><span>392.5 389.8167994835544
</span><span>15.7 65.88576964183908
</span><span>48.8 45.426967757099156
</span><span>113.0 99.98377278307227
</span><span>48.7 52.246568385345796
</span><span>40.3 38.607367128852516
</span><span>59.6 76.11517058420904
</span><span>152.8 147.72097718079874
</span><span>73.4 161.360178437292
</span><span>21.3 59.066169013592436
</span><span>55.6 48.83676807122248
</span><span>194.5 123.85237498193551
</span></code></pre>
<h2 id="error">5.6. How good is our model:</h2>
<p>We need to able to measure how good our model is. Among the methods available to achieve this task, let's use the most common and the most used: the <a href="#">Root mean squared error</a>.</p>
<h3 id="rmse">RMSE:</h3>
<p>One way (the most common way?) to assess how well our model fits our data set is to calculate the Root Mean Squared Error. The RMSE is:</p>
<ul>
<li>the square root</li>
<li>of the mean</li>
<li>of the square of all of the errors</li>
</ul>
<p>Let's write a function to calculate the RMSE:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>rmse(actual: t.Sequence[float], predicted: t.Sequence[float]) -> float:
</span><span> sum_error = </span><span style="color:#b5cea8;">0.0
</span><span> </span><span style="color:#569cd6;">for </span><span>i </span><span style="color:#569cd6;">in </span><span>range(len(actual)):
</span><span> prediction_error = predicted[i] - actual[i]
</span><span> sum_error += prediction_error**</span><span style="color:#b5cea8;">2
</span><span> mean_error = sum_error / float(len(actual))
</span><span> </span><span style="color:#569cd6;">return </span><span>math.sqrt(mean_error)
</span></code></pre>
<p>If we develop mutliple models, and we want to verify which one performs better:</p>
<ul>
<li>Calculate the RMSE value for each model</li>
<li>The lower the RMSE value, the better the model.</li>
<li>A RMSE value of 0 indicate a perfect fit.</li>
</ul>
<p>The data set we're working with has a range: [Y_min, Y_max]. This range is important in determining whether a given RMSE value is "low" or "high". There is no "good" or no "bad" RMSE value. It always depends on the range of the data set we're working with.</p>
<h3 id="normalized-rmse">Normalized RMSE:</h3>
<p>The NRMSE is the RMSE divided by Y_max - Y_min. The result will be between 0 and 1, where values closer to 0 represent better fitting models. The normalization facilitates the comparison between data sets with different scales:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>rmse(
</span><span> actual: t.Sequence[float], predicted: t.Sequence[float]
</span><span>) -> t.Tuple[float, float]:
</span><span> mean_error = sum([(predicted[i] - actual[i])**</span><span style="color:#b5cea8;">2 </span><span style="color:#569cd6;">for </span><span>i </span><span style="color:#569cd6;">in </span><span>range(len(actual))]) / float(len(actual))
</span><span> rmse = math.sqrt(mean_error)
</span><span> nrmse = rmse / (max(predicted) - min(predicted))
</span><span> </span><span style="color:#569cd6;">return </span><span>rmse, nrmse
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>>> python3 script.py
</span><span>The RMSE is 23.27666203730159
</span><span>Normalized RMSE is 0.06371542339113609
</span><span>
</span></code></pre>
<h2 id="final-plot">5.7. Regression line:</h2>
<center>
<img src="https://nskm.xyz/processed_images/plot_regression_line.156ebbadc8a6d8e7.png" />
</center>
<br>
<br>
<br>
<h1 id="class">The complete program:</h1>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>math
</span><span style="color:#9b9b9b;">import </span><span>random
</span><span style="color:#9b9b9b;">import </span><span>requests </span><span style="color:#608b4e;"># type:ignore
</span><span style="color:#9b9b9b;">import </span><span>matplotlib </span><span style="color:#9b9b9b;">as </span><span>mpl </span><span style="color:#608b4e;"># type:ignore
</span><span style="color:#9b9b9b;">import </span><span>matplotlib.pyplot </span><span style="color:#9b9b9b;">as </span><span>plt </span><span style="color:#608b4e;"># type:ignore
</span><span style="color:#9b9b9b;">import </span><span>typing </span><span style="color:#9b9b9b;">as </span><span>t
</span><span style="color:#9b9b9b;">import </span><span>pathlib
</span><span style="color:#9b9b9b;">from </span><span>dataclasses </span><span style="color:#9b9b9b;">import </span><span>dataclass, field
</span><span>
</span><span>
</span><span style="color:#608b4e;"># setup plots
</span><span>fig: mpl.figure.Figure
</span><span>ax: mpl.axes.Axes
</span><span>
</span><span>fig, ax = plt.subplots()
</span><span>ax.set_xlabel(</span><span style="color:#d69d85;">"X-coordinates "</span><span>)
</span><span>ax.set_ylabel(</span><span style="color:#d69d85;">"Y-coordinates"</span><span>)
</span><span>ax.set_title(</span><span style="color:#d69d85;">"Graphical representation of our points"</span><span>)
</span><span>ax.grid(</span><span style="color:#569cd6;">True</span><span>)
</span><span>
</span><span>splits = t.Tuple[
</span><span> t.Collection[float], t.Collection[float], t.Collection[float], t.Collection[float]
</span><span>]
</span><span>
</span><span>
</span><span>@dataclass
</span><span style="color:#569cd6;">class </span><span>SimpleLinearRegression:
</span><span> filename: t.Union[pathlib.Path, str] = </span><span style="color:#d69d85;">"./dataset.txt"
</span><span> __X: t.Collection[float] = field(init=</span><span style="color:#569cd6;">False</span><span>)
</span><span> __Y: t.Collection[float] = field(init=</span><span style="color:#569cd6;">False</span><span>)
</span><span> X_train: t.Collection[float] = field(init=</span><span style="color:#569cd6;">False</span><span>)
</span><span> Y_train: t.Collection[float] = field(init=</span><span style="color:#569cd6;">False</span><span>)
</span><span> X_test: t.Collection[float] = field(init=</span><span style="color:#569cd6;">False</span><span>)
</span><span> Y_test: t.Collection[float] = field(init=</span><span style="color:#569cd6;">False</span><span>)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>__post_init__(self) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> </span><span style="color:#569cd6;">with </span><span>open(self.filename, </span><span style="color:#d69d85;">"r"</span><span>) </span><span style="color:#569cd6;">as </span><span>f:
</span><span> slines = [
</span><span> line.strip().split(</span><span style="color:#d69d85;">"</span><span style="color:#e3bbab;">\t</span><span style="color:#d69d85;">"</span><span>) </span><span style="color:#569cd6;">for </span><span>line </span><span style="color:#569cd6;">in </span><span>[li </span><span style="color:#569cd6;">for </span><span>li </span><span style="color:#569cd6;">in </span><span>f.readlines()[</span><span style="color:#b5cea8;">1</span><span>:]]
</span><span> ]
</span><span> nlines = [(float(x), float(y)) </span><span style="color:#569cd6;">for </span><span>x, y </span><span style="color:#569cd6;">in </span><span>slines]
</span><span> self.__X, self.__Y = zip(*nlines)
</span><span> self.__split_train_test()
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>scatter_plot(self, ax: mpl.axes.Axes, color: str = </span><span style="color:#d69d85;">"red"</span><span>, label=</span><span style="color:#569cd6;">None</span><span>) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> ax.scatter(self.__X, self.__Y, color=color, label=label)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>line_plot(self, ax: mpl.axes.Axes) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> b, a = self.estimate_coefficients()
</span><span> x = [</span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#b5cea8;">130</span><span>]
</span><span> y = [a * x + b </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>x]
</span><span> ax.plot(x, y, color=</span><span style="color:#d69d85;">"green"</span><span>)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>__mean(self, lst: t.Collection[float]) -> float:
</span><span> </span><span style="color:#569cd6;">return </span><span>sum(lst) / len(lst)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>__var(self, lst: t.Collection[float]) -> float:
</span><span> m = self.__mean(lst)
</span><span> </span><span style="color:#569cd6;">return </span><span>sum([(x - m) ** </span><span style="color:#b5cea8;">2 </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>lst]) / len(lst)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>__cov(self, X: t.Sequence[float], Y: t.Sequence[float]) -> float:
</span><span> mean_x = self.__mean(X)
</span><span> mean_y = self.__mean(Y)
</span><span> covar = </span><span style="color:#b5cea8;">0.0
</span><span> </span><span style="color:#569cd6;">for </span><span>i </span><span style="color:#569cd6;">in </span><span>range(len(X)):
</span><span> covar += (X[i] - mean_x) * (Y[i] - mean_y)
</span><span> </span><span style="color:#569cd6;">return </span><span>covar / len(X)
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>__split_train_test(self, frac: float = </span><span style="color:#b5cea8;">0.2</span><span>) -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> indexes = list(range(len(self.__X)))
</span><span> indexes_test = set(random.sample(indexes, int(frac * len(self.__X))))
</span><span>
</span><span> </span><span style="color:#608b4e;"># get 20% of Xs based on the indexes, do the same for Ys
</span><span> self.X_test = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(self.__X) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">in </span><span>indexes_test]
</span><span> self.Y_test = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(self.__Y) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">in </span><span>indexes_test]
</span><span>
</span><span> </span><span style="color:#608b4e;"># get 80% of Xs based on the indexes, do the same for Ys
</span><span> self.X_train = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(self.__X) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">not in </span><span>indexes_test]
</span><span> self.Y_train = [n </span><span style="color:#569cd6;">for </span><span>i, n </span><span style="color:#569cd6;">in </span><span>enumerate(self.__Y) </span><span style="color:#569cd6;">if </span><span>i </span><span style="color:#569cd6;">not in </span><span>indexes_test]
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>estimate_coefficients(
</span><span> self,
</span><span> ) -> t.Tuple[float, float]:
</span><span> mean_x, mean_y = self.__mean(self.X_train), self.__mean(self.Y_train)
</span><span> b1 = self.__cov(self.X_train, self.Y_train) / self.__var( </span><span style="color:#608b4e;"># type:ignore
</span><span> self.X_train
</span><span> )
</span><span> b0 = mean_y - b1 * mean_x
</span><span> </span><span style="color:#569cd6;">return </span><span>b0, b1
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>rmse(self) -> t.Tuple[float, float]:
</span><span> b, a = self.estimate_coefficients()
</span><span> predicted = [a * x + b </span><span style="color:#569cd6;">for </span><span>x </span><span style="color:#569cd6;">in </span><span>self.X_test]
</span><span> actual = self.Y_test
</span><span> mean_error = sum(
</span><span> [(predicted[i] - actual[i]) ** </span><span style="color:#b5cea8;">2 </span><span style="color:#569cd6;">for </span><span>i </span><span style="color:#569cd6;">in </span><span>range(len(actual))] </span><span style="color:#608b4e;"># type:ignore
</span><span> ) / float(
</span><span> len(actual)
</span><span> )
</span><span> rmse = math.sqrt(mean_error)
</span><span> nrmse = math.sqrt(mean_error) / (max(predicted) - min(predicted))
</span><span> </span><span style="color:#569cd6;">return </span><span>rmse, nrmse
</span><span>
</span><span>
</span><span style="color:#608b4e;"># program entry point
</span><span style="color:#569cd6;">if </span><span>__name__ == </span><span style="color:#d69d85;">"__main__"</span><span>:
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>main() -> </span><span style="color:#569cd6;">None</span><span>:
</span><span> model = SimpleLinearRegression()
</span><span> model.scatter_plot(ax)
</span><span> model.line_plot(ax)
</span><span> b, a = model.estimate_coefficients()
</span><span> rmse, nrmse = model.rmse()
</span><span>
</span><span> print(</span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"The slope is </span><span>{a}</span><span style="color:#d69d85;">, the intercept is </span><span>{b}</span><span style="color:#d69d85;">"</span><span>)
</span><span> print(</span><span style="color:#569cd6;">f</span><span style="color:#d69d85;">"The RMSE and NRMSE are: </span><span>{rmse}</span><span style="color:#d69d85;"> and </span><span>{nrmse}</span><span style="color:#d69d85;">"</span><span>)
</span><span> plt.show()
</span><span>
</span><span> main()
</span><span>
</span></code></pre>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>The slope is 3.426108589853597, the intercept is 20.478117458572427
</span><span>The RMSE and NRMSE are: 27.82841276749733 and 0.07520791645397
</span></code></pre>
<br>
<h1 id="statistics">Another solution, using the statistics module:</h1>
<p>Before jumping to big and powerful libraries such as <a href="https://numpy.org/doc/stable/reference/routines.statistics.html">NumPy</a>, <a href="https://docs.scipy.org/doc/scipy/reference/stats.html">SciPy</a>, <a href="https://github.com/pandas-dev/pandas">Pandas</a>, <a href="https://github.com/raphaelvallat/pingouin">Pingouin</a>, <a href="https://github.com/statsmodels/statsmodels">Statsmodel</a>, and all the others. We have to notice, Python provides us with a module named <a href="https://https://docs.python.org/3/library/statistics.html">statistics</a>. Even though the statistics module is not intended to be a competitor to the ones listed above, it's interesting to see that inside, there is:</p>
<ul>
<li>a <a href="https://docs.python.org/3/library/statistics.html#statistics.mean">mean</a> function</li>
<li>a <a href="https://docs.python.org/3/library/statistics.html#statistics.variance">variance</a> function</li>
<li>a <a href="https://docs.python.org/3/library/statistics.html#statistics.covariance">covariance</a> function</li>
<li>even a <a href="https://https://docs.python.org/3/library/statistics.html#statistics.linear_regression">linear regression</a> function.</li>
</ul>
<p>Our program will suddenly become shorter:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>statistics </span><span style="color:#9b9b9b;">as </span><span>lestat
</span><span>
</span><span style="color:#608b4e;"># load dataset
</span><span>X, Y = read_text(</span><span style="color:#d69d85;">"dataset.txt"</span><span>)
</span><span>
</span><span style="color:#608b4e;"># extract coefficients
</span><span>a, b = lestat.linear_regression(X, Y)
</span></code></pre>
<br>
<h1 id="code">Source code:</h1>
<p><a href="https://c.tenor.com/a5n5GalczLsAAAAM/relief-phew.gif"><em>Phew</em></a>. All the cool devs today will share the source code via a Jupyter notebook stored on Github or on Colab. According to the <a href="https://sfconservancy.org/">Software Freedom Conservancy</a>, we should all <a href="https://sfconservancy.org/GiveUpGitHub/">Give Up GitHub</a>. And as a free software advocate, I agree with them. The source code can be found <a href="/assets/script.py">here</a> and the data set <a href="/assets/dataset.txt">here</a>. You're welcome.</p>
<br>
<h1 id="more-on-this-topic">More on this topic:</h1>
<p>I really hope you've learned something, really. Talking about the whole subject won't be possible in this tiny article. To dive deeper and learn even more, let me please recommend the following links:</p>
<ul>
<li><a href="https://pypi.org/project/datasette/">Datasette</a></li>
<li><a href="https://matplotlib.org/">Matplotlib</a></li>
<li><a href="https://xkcd.com/378/">Real programmers</a></li>
<li><a href="https://www.sciencedirect.com/topics/engineering/root-mean-squared-error">Root-Mean-Squared Error</a></li>
<li><a href="https://posit.co/">RStudio is becoming Posit</a></li>
<li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html">Ordinary least squares Linear Regression with scikit learn</a></li>
</ul>
Wall of Fame2022-07-04T00:00:00+00:002022-07-04T00:00:00+00:00
Unknown
https://nskm.xyz/wall/<p>Here it is, the Wall of Fame.</p>
<div>
<a href="https://nskm.xyz/wof/pictures/HOF_a_core_dev_followed_me.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_a_core_dev_followed_me.8e8245579593b0f8.jpg" alt="foo" title="a core dev followed me">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_ashwinvis_enjoyed_my_mypy_article.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_ashwinvis_enjoyed_my_mypy_article.574457f975fea21d.png" alt="foo" title="ashwinvis enjoyed my mypy article">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Bash_aliases_and_functions.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Bash_aliases_and_functions.3ca5c8e249e0a455.png" alt="foo" title="Bash aliases and functions">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Current_status.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Current_status.d406aaba585359c1.jpg" alt="foo" title="Current status">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_customized_Hugo_hermit_theme.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_customized_Hugo_hermit_theme.8ae1ad770e1b0734.png" alt="foo" title="customized Hugo hermit theme">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Django_donator.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Django_donator.bef05b8e403fe16a.png" alt="foo" title="Django donator">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Djibril_being_grateful_to_me.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Djibril_being_grateful_to_me.85ea81234fa4fcbd.png" alt="foo" title="Djibril being grateful to me">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_donkirby_reusing_my_work.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_donkirby_reusing_my_work.b8f37fd3c4e90ef5.png" alt="foo" title="donkirby reusing my work">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Dont_introduce_yourself.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Dont_introduce_yourself.6772b572b264769c.png" alt="foo" title="Dont introduce yourself">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Efficacite_de_la_creation.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Efficacite_de_la_creation.f366ac3a8200fb9d.png" alt="foo" title="Efficacite de la creation">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Emacs_massive_find_replace.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Emacs_massive_find_replace.1e9e80f82f0c3d1c.png" alt="foo" title="Emacs massive find replace">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Exploring_PyPolars.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Exploring_PyPolars.41f3d0e4b395fc6d.png" alt="foo" title="Exploring PyPolars">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Figthing_against_Mypy.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Figthing_against_Mypy.de7f548ae09b45f4.png" alt="foo" title="Figthing against Mypy">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_First_day_on_Mastodon.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_First_day_on_Mastodon.38d18dd00f5e2a42.png" alt="foo" title="First day on Mastodon">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Gandi_API.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Gandi_API.2e4372b1327a02b5.png" alt="foo" title="Gandi API">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Gandi_API_again.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Gandi_API_again.847b2968e784e92c.png" alt="foo" title="Gandi API again">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Gnus_once_upon_a_time.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Gnus_once_upon_a_time.8dac3ebd9b08dca3.png" alt="foo" title="Gnus once upon a time">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Hercules.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Hercules.8a20d66567588c01.png" alt="foo" title="Hercules">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_I_am_a_libertarian_leftist.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_I_am_a_libertarian_leftist.e9c7235b551f63e7.png" alt="foo" title="I am a libertarian leftist">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Julien_Palard_reacted_to_my_toot.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Julien_Palard_reacted_to_my_toot.02c9b51bdf84c818.png" alt="foo" title="Julien Palard reacted to my toot">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_kingbuzzman_reused_my_work.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_kingbuzzman_reused_my_work.8584a39a6c706df8.png" alt="foo" title="kingbuzzman reused my work">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Konami_code.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Konami_code.6edf6e64efe8be15.png" alt="foo" title="Konami code">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Logilab_thanks_me_for_my_work.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Logilab_thanks_me_for_my_work.374fce72f1aadf39.png" alt="foo" title="Logilab thanks me for my work">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_me.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_me.4e54225d2209fa39.jpg" alt="foo" title="me">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Mercurial_aliases.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Mercurial_aliases.7e0820558947e9a7.png" alt="foo" title="Mercurial aliases">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Never_knew_I_was_one_of_the_creators_of_Python_Idle.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Never_knew_I_was_one_of_the_creators_of_Python_Idle.538d0f244bd173d0.png" alt="foo" title="Never knew I was one of the creators of Python Idle">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Playing_with_Delta_Chat.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Playing_with_Delta_Chat.a3710e84afed5bb2.png" alt="foo" title="Playing with Delta Chat">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Playing_with_Git_hooks.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Playing_with_Git_hooks.d0e265d8ee5507c9.png" alt="foo" title="Playing with Git hooks">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Playing_with_Quarto.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Playing_with_Quarto.b1953381cb548b9a.png" alt="foo" title="Playing with Quarto">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_PR_accepted_to_typeshed_repo.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_PR_accepted_to_typeshed_repo.c7b86067ce4b5ac5.png" alt="foo" title="PR accepted to typeshed repo">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Proud_Emacs_Stackexchange_User.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Proud_Emacs_Stackexchange_User.c664f068e88bbb13.png" alt="foo" title="Proud Emacs Stackexchange User">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Proud_Emacs_user.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Proud_Emacs_user.4ea7bae1980d4fda.jpg" alt="foo" title="Proud Emacs user">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_PSF_member_yes.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_PSF_member_yes.8a574b3de6c0aed4.png" alt="foo" title="PSF member yes">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_pycon_africa_speaker.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_pycon_africa_speaker.9bfe680412266c97.jpg" alt="foo" title="pycon africa speaker">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Python_release_meassages_are_the_bests.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Python_release_meassages_are_the_bests.f36dffb844ed6e2f.png" alt="foo" title="Python release meassages are the bests">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Python_subprocess_example.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Python_subprocess_example.64a643f6d69895bb.png" alt="foo" title="Python subprocess example">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Reading.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Reading.e1323ecf5a19501c.jpg" alt="foo" title="Reading">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Rules_for_a_good_life.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Rules_for_a_good_life.d6c4185e1ab08970.jpg" alt="foo" title="Rules for a good life">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_sqlite_select_as_CSV.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_sqlite_select_as_CSV.1f30c9897fc1bb83.png" alt="foo" title="sqlite select as CSV">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Taking_stance.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Taking_stance.a14757f3246d4612.png" alt="foo" title="Taking stance">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Time_to_upgrade_distribution.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Time_to_upgrade_distribution.8e5e464c1d4dad81.png" alt="foo" title="Time to upgrade distribution">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_When_I_was_dealing_with_Paypox_wtf.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_When_I_was_dealing_with_Paypox_wtf.13ea33a6beb67dd2.png" alt="foo" title="When I was dealing with Paypox wtf">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_When_I_was_working_on_Vialegal.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_When_I_was_working_on_Vialegal.a471c4021dd87533.png" alt="foo" title="When I was working on Vialegal">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Why_I_dont_use_social.media.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Why_I_dont_use_social.media.a93363674756712b.png" alt="foo" title="Why I dont use social.media">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_Yasnippets_to_send_a_resume.png" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_Yasnippets_to_send_a_resume.9f43176f3144c85c.png" alt="foo" title="Yasnippets to send a resume">
</a>
<a href="https://nskm.xyz/wof/pictures/HOF_You_are_your_own_timezone.jpg" target="_blank">
<img src="https://nskm.xyz/processed_images/HOF_You_are_your_own_timezone.ebbd3ce55e8bf6ff.jpg" alt="foo" title="You are your own timezone">
</a>
</div>
Writing2022-05-20T00:00:00+00:002022-05-20T00:00:00+00:00
Unknown
https://nskm.xyz/posts/blg/<p>I often receive resumes from people looking for a job and asking me for help. To all of them I always ask: "Do you have a website ?", "Do you write ?". And most of the time, the answer is negative.</p>
<p>Writing skills are important because they allow the reader to get their first impression of who is the writer. For example: writing skills displayed in resume, cover letter email communications, website. In my humble opinion, if you're looking for a job as a software engineer, you should have a tiny space on the internet. A space in which you write about what you experiment, what you code, what you build.</p>
<p>This article is an attempt to answer, why I think everybody (every software engineer) should have a website. Or at least, why everybody should write. This article is also <em>kind of</em> a recap of a talk I gave for the <a href="https://dit.sn">Dakar Institute of Tecnology</a>.</p>
<center>
<img src="https://nskm.xyz/processed_images/wr.45c3c0f8e7026afa.jpg" />
</center>
<br>
<br>
<h1 id="some-reasons-for-you-to-start-writing"><u>Some reasons for you to start writing: </u></h1>
<h2 id="1-improve-your-knowledge-on-a-topic"><em>1. Improve your knowledge on a topic</em></h2>
<p>You'll often have to write about subjects that you are not familiar with, so finding quality sources quickly is a skill you'll develop. Before you write a single word, you need to do your research about the topic you're writing on. The more you do research, the more you find quality sources, the more you improve your knowledge.</p>
<h2 id="2-organize-your-thoughts"><em>2. Organize your thoughts</em></h2>
<p>You can write down more than you can easily remember, so that your capacity to consider a number of ideas at the same time is broadened. Once those ideas are written down, you can move them around, change them. You can also reject those that are finally not good enough. You will stay with good, original clearly defined ideas. And you will be able to organize and communicate them easily.</p>
<h2 id="3-share-your-expertise-meet-fellow-specialist-and-grow-your-knowledge"><em>3. Share your expertise, meet fellow specialist and grow your knowledge</em></h2>
<p>Knowledge is the only thing that grows when shared. Sharing your knowledge helps you grow your knowledge by doing research, synthesizing multiple and different viewpoints. Sharing your expertise means inviting people to read you, inviting people to a new conversation. A conversation is always an occasion to meet new people and to grow as a human.</p>
<h2 id="4-make-money"><em>4. Make money</em></h2>
<p>Even professions that don't focus on writing require written communication skills. The tech industry for example, is more and more looking for people who are able to write. Those writings include things like instruction manuals, user manuals, journal articles, product descriptions, memos, wikis, documentations, etc...</p>
<p>You can have a look :</p>
<ul>
<li><a href="https://www.indeed.com/q-Technical-Writer-jobs.html?vjk=1e9cd66778e25548">here</a></li>
<li><a href="https://www.upwork.com/freelance-jobs/technical-writing/">then here</a></li>
<li><a href="https://www.glassdoor.com/Job/technical-writer-jobs-SRCH_KO0,16.htm">here again</a></li>
<li><a href="https://careers.google.com/jobs/results/93728821645058758-technical-writer-google-cloud/?distance=50&q=technical%20writer">also here</a></li>
</ul>
<p>By the way:</p>
<ul>
<li><a href="https://realpython.com/write-for-us/">Real Python is hiring</a>.</li>
<li><a href="https://lwn.net/Articles/895695/">LWN is hiring too</a></li>
</ul>
<p><em>disclaimer: I'm not paid by anybody to share those job offers</em></p>
<br>
<h1 id="what-are-the-topic-to-write-about"><u>What are the topic to write about: </u></h1>
<p>Coming up with writing inspiration is tough. My advice would be, <strong>in all the cases, always write about something you really like, or something you really enjoy.</strong> As a technical writer, you can write different kinds of articles:</p>
<h2 id="1-story"><em>1. Story</em></h2>
<p>Human beings are interested in stories about people. If your reader knows you a little through your blog, the reader will remember who you are and the reader will be more interested by your technical articles. <a href="https://news.ycombinator.com/ask">Here</a> you'll find different kinds of stories.</p>
<h2 id="2-opinion"><em>2. Opinion</em></h2>
<p>Giving an opinion is a common way of interacting with other people. Of course you don't simply share an opinion, you'll also share the compelling and interesting argument that support this opinion. Also, please don't spend your time attacking others people or spreading controversial ideas. Here is an <a href="https://drewdevault.com/2019/12/09/Developers-shouldnt-distribute.html">axample</a> of a shared opinion.</p>
<h2 id="3-how-tos"><em>3. HOW-TOs</em></h2>
<p>An how-to is a document providing detailed and practical step-by-step guide on how to do something. In a precise, step-by-step approach, you should walk your reader through the process. You should make sure your reader can reproduce the same results by following the provided steps. <a href="https://docs.python.org/3/howto/pyporting.html">Example</a> of an HOWTO.</p>
<h2 id="4-how-something-work"><em>4. How something work</em></h2>
<p>There are people <em>like me</em> who want to know how almost everything works. This kind of article is suitable when you want to deep dive into a very specific topic. For example, somebody already using Django sessions, may be suddenly wondering how does Django sessions work under the hood ? Here is an <a href="https://eli.thegreenplace.net/2011/06/24/how-django-sessions-work-introduction">article</a> showing how Django sessions work.</p>
<h2 id="5-reviews"><em>5. Reviews</em></h2>
<p>You have read, used, tested, played with something. In a review, your attempt will be to write about what was your experience. You'll describe the thing, you'll share what were your feelings about the thing, you'll show the strengths and the weaknesses of the thing. Example of <a href="https://emptysqua.re/blog/review-of-autotools-a-practitioners-guide/">review</a>.</p>
<br>
<h1 id="identifying-the-target-audience"><u>Identifying the target audience: </u></h1>
<p>Identifying your intended audience will help you write effective content that achieves your goals. What are you writing ? Who are you writing for ? Knowing your audience will help you adapt your content accordingly. And if you don't know who to write for, then the following is a serie of questions you can ask yourself:</p>
<ul>
<li>Who am I writing for ?</li>
<li>What subject am I writing about ?</li>
<li>What does my audience already know about the subject?</li>
<li>What does my audience want to know about the subject?</li>
<li>What are the questions will my audience have?</li>
</ul>
<br>
<h1 id="finding-the-audience"><u>Finding the audience: </u></h1>
<p>For example, <strong>as a Python specialist</strong>, the audience can be found in different places. Pycoder weekly news want to hear from you about projects you are working on, conferences you are running, and articles you want to <a href="https://pycoders.com/submissions">share</a>.</p>
<p>Awesome Python is another newsletter who want you to <a href="https://python.libhunt.com/contribute">contribute</a> links to blog posts, tutorials, libraries, events, videos, books.</p>
<p>You can als share your posts on <a href="https://lobste.rs/">Lobsters</a>, on <a href="https://tilde.news/">Tilde News</a>, on <a href="https://dev.to/">dev.to</a>, on <a href="https://towardsdatascience.com/about">towardsdatascience</a>.</p>
<br>
<h1 id="improving-your-writing"><u>Improving your writing: </u></h1>
<p>The following is what I recommend. In all the cases, you should find what is best for you.</p>
<h2 id="1-practice"><em>1. Practice</em></h2>
<p>The most effective writers write every day, <em>at least a bit</em>. The more you write, the more your brain form connections between neurons. Those connections get stronger the more often you write. After some time and regular practice, you no longer have to think about it. This is when writing becomes a habit.</p>
<h2 id="2-make-time"><em>2. Make time</em></h2>
<p>Prepare to spend between 90 minutes and 3 hours writing. But hey, 15 minutes can be useful if you do it every day. Don't fool yourself, thinking you'll wait for a big chunk of free time to start. That will never happen.</p>
<h2 id="3-when"><em>3. When</em></h2>
<p>You are much smarter and more resilient after you have slept properly and ate. Again, find what suits you best. According to some studies, creativity peaks in the morning. My experience tells me that's true.</p>
<h2 id="4-frequency"><em>4. Frequency</em></h2>
<p>Write only when you feel it.</p>
<br>
<h1 id="tools-for-writing-ssg"><u>Tools for writing... SSG:</u></h1>
<p>You should provide yourself with the right technology. Here again, I can tell you what I do, but there are plenty of differents tools, you <a href="https://app.hackernoon.com/signup">should</a> <a href="https://pages.github.com/">always</a> <a href="https://jekyllrb.com/">find</a> <a href="https://substack.com/for-bloggers">what</a> <a href="https://write.as/start">suits</a> <a href="https://www.blogger.com/about/?bpli=1">you</a> <a href="https://medium.com/creators">best</a>.</p>
<p>I've choosen to build my website using a Static Site Generator (SSG). An ssg is a tool that generates a full static HTML website based on raw data (text files) and a set of templates (html, css, js files). :</p>
<center>
<img src="https://nskm.xyz/processed_images/ssg2.3f40eef587b8186f.png" />
</center>
<p style="text-align:center"> <a href="https://devopedia.org/static-site-generators">Image source</a> </p>
<br>
<h1 id="why-ssg"><u>Why SSG:</u></h1>
<ul>
<li>No need for internet for writing, no need to login somewhere before writing.</li>
<li>no need for a particular environment, just a text editor, and a browser.</li>
<li>No vendor lock in, no dependency on any product for writing or migrating your content.</li>
<li>No need for a program to dynamically generate webpages. Web pages load faster.</li>
<li>No dependencies running on the server, just a web server. Increased security</li>
</ul>
<br>
<h1 id="i-m-convinced-can-you-show-me-an-example-please"><u>I'm convinced, can you show me an example please:</u></h1>
<p>The <a href="https://nskm.xyz/posts/blg/">article</a> you're reading right now, was generated from a <a href="https://bin.nskm.xyz/pastes/markdown.md">markdown file</a> using an SSG named <a href="https://www.getzola.org/">Zola</a> and a theme named <a href="https://github.com/barlog-m/oceanic-zen">Oceanic Zen</a>.</p>
<br>
<br>
<p>To finish this article, I would like to quote <a href="https://en.wikipedia.org/wiki/Leslie_Lamport">Leslie Lamport</a></p>
<br>
<blockquote>
<p><strong>"If you’re thinking without writing, you only think you’re thinking."</strong></p>
</blockquote>
<blockquote>
<p>— Leslie Lamport</p>
</blockquote>
<br>
<h1 id="links">Links:</h1>
<p>To learn more on this topic, I strongly recommend the following links:</p>
<ul>
<li><a href="https://www.yieldcode.blog/post/why-engineers-should-write/">Why engineers should focus on writing</a></li>
<li><a href="https://yewtu.be/watch?v=RnY5iJea5ww&t=77s">Think and write, with Leslie Lamport</a></li>
<li><a href="https://yewtu.be/watch?v=HOXwDWCoqQg">Jordan Peterson on the importance of reading and writing</a></li>
<li><a href="https://yewtu.be/watch?v=vyVpRiqOvt4">How writing online made me a millionaire</a></li>
<li><a href="https://yewtu.be/watch?v=tod1sraj8we">How to improve your clarity of thought ("writing is thinking")</a></li>
<li><a href="https://yewtu.be/watch?v=fxLFjOa-9UY">Programmers that don't blog should start right now</a></li>
<li><a href="https://yewtu.be/watch?v=p8dve3MqQW4">Benefits of writing</a></li>
<li><a href="https://yewtu.be/watch?v=mmU25Xd0BGs">Technical blogging for Python programmers</a></li>
<li><a href="https://yewtu.be/watch?v=aTgPJQ9Dy7Q">Look deeper -- write -- the wonders of writing</a></li>
<li><a href="https://yewtu.be/watch?v=t4vKPhjcMZg">What nobody tells you about documentation</a></li>
<li><a href="https://en.wikipedia.org/wiki/Verba_volant,_scripta_manent">Spoken words fly away, written words remain</a></li>
<li><a href="https://en.wikipedia.org/wiki/Circadian_rhythm">Circadian rythm</a></li>
</ul>
<h1 id="links-again">Links again:</h1>
<ul>
<li><a href="https://jamstack.org/generators/">Static Site Generators</a></li>
<li><a href="https://about.gitlab.com/blog/2022/04/18/comparing-static-site-generators/">How to choose the right static site generator</a></li>
<li><a href="https://www.getzola.org/">Zola, your one-stop static site engine</a></li>
<li><a href="https://getpublii.com/">Publii, Static Site Generator with GUI</a></li>
<li><a href="https://gohugo.io/">Hugo, the world’s fastest framework for building websites</a></li>
<li><a href="https://git.disroot.org/orbifx/logarion">Logarion, a suite of tools, for discovering, collecting & exchanging texts</a></li>
</ul>
<h1 id="even-more-links">Even more links:</h1>
<p>Things that were discussed during the talk:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Literate_programming">Literate programming</a></li>
<li><a href="https://yewtu.be/watch?v=Mr3WTR0a5SM">Knuth on Literate Programming</a></li>
<li><a href="https://orgmode.org/worg/org-contrib/babel/">Babel: active code in Org-mode</a></li>
<li><a href="https://pandoc.org/">Pandoc, a universal document converter</a></li>
<li><a href="https://quarto.org/">Quarto, open-source scientific and technical publishing system built on Pandoc</a></li>
</ul>
<br>
<center>
<img src="https://nskm.xyz/processed_images/wr2.c2d34d34c2abf8e1.jpg" />
</center>
Function pointers2022-04-29T16:15:50+00:002022-04-29T16:15:50+00:00
Unknown
https://nskm.xyz/posts/fp/<p>In Python, we say, functions are <em>first class citizen</em>. This means, functions can be used like any other objects. They can be stored inside variables, they can be passed as argument to a function, they can be returned from a function. The following is a very basic example:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>square = </span><span style="color:#569cd6;">lambda </span><span>x: x * x
</span><span>cube = </span><span style="color:#569cd6;">lambda </span><span>x: x * x * x
</span><span>
</span><span>operations = {</span><span style="color:#b5cea8;">1</span><span>: square, </span><span style="color:#b5cea8;">2</span><span>: cube}
</span><span>choice = int(input(</span><span style="color:#d69d85;">"Square (1) or Cube (2) ?: "</span><span>))
</span><span>value = int(input(</span><span style="color:#d69d85;">"Value ?: "</span><span>))
</span><span>
</span><span>print(operations[choice](value))
</span></code></pre>
<br>
<p>Let's start with a very simple function to print out a simple "hello world" using a function pointer:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include</span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span style="color:#608b4e;">// defining a function ptr
</span><span>void (</span><span style="color:#569cd6;">*</span><span>greet_ptr)();
</span><span>
</span><span style="color:#608b4e;">// a function taking no argument and returning nothing
</span><span style="color:#569cd6;">void </span><span>greet(){
</span><span> printf(</span><span style="color:#d69d85;">"Hello world!</span><span style="color:#e3bbab;">\n\n</span><span style="color:#d69d85;">"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main(){
</span><span> </span><span style="color:#608b4e;">// make our function pointer points to greet function
</span><span> greet_ptr = </span><span style="color:#569cd6;">&</span><span>greet;
</span><span>
</span><span> </span><span style="color:#608b4e;">// call greet function via the function pointer
</span><span> greet_ptr();
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span><span>
</span></code></pre>
<br>
<h3 id="slowly-please"><strong>Slowly please?</strong></h3>
<p>The first interesting line is:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span>void (</span><span style="color:#569cd6;">*</span><span>greet_ptr)();
</span></code></pre>
<ol>
<li>We're defining a <em>pointer to a function</em>, named <strong>greet_ptr</strong>.</li>
<li>We're using the <strong>*</strong> notation to signify that it is a pointer.</li>
<li>The function pointed to, should receive no argument: <strong>()</strong></li>
<li>The function should return nothing: <strong>void</strong>.</li>
<li>We need parenthesis, otherwise it becomes <strong>void *greet_ptr()</strong>, a function returning a <strong>void pointer.</strong></li>
</ol>
<br>
<p>The next interesting lines are:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span>greet_ptr = </span><span style="color:#569cd6;">&</span><span>greet;
</span><span>greet_ptr();
</span></code></pre>
<ol>
<li><code>greet_ptr</code> is a pointer, so it should receive <em>the address</em> of a function.</li>
<li><code>greet</code> is a function, taking no argument, and returning void.</li>
<li><code>greet_ptr()</code>: let's execute the code living at the address stored by greet_ptr.</li>
</ol>
<br>
<h3 id="what-if-the-function-pointed-to-takes-arguments"><strong>What if the function pointed to, takes arguments?</strong></h3>
<p>Simple example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include</span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span style="color:#608b4e;">// defining a function ptr
</span><span>void (</span><span style="color:#569cd6;">*</span><span>greet_ptr)(</span><span style="color:#569cd6;">char *</span><span>);
</span><span>
</span><span style="color:#569cd6;">void </span><span>greet(</span><span style="color:#569cd6;">char *</span><span>name){
</span><span> printf(</span><span style="color:#d69d85;">"Hello </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">!</span><span style="color:#e3bbab;">\n\n</span><span style="color:#d69d85;">"</span><span>, name);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main(){
</span><span> </span><span style="color:#608b4e;">// make our function pointer points to greet function
</span><span> greet_ptr = </span><span style="color:#569cd6;">&</span><span>greet;
</span><span>
</span><span> </span><span style="color:#608b4e;">// call greet function via the function pointer
</span><span> greet_ptr(</span><span style="color:#d69d85;">"foo"</span><span>);
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span></code></pre>
<br>
<p>The first interesting line is:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span>void (</span><span style="color:#569cd6;">*</span><span>greet_ptr)(</span><span style="color:#569cd6;">char *</span><span>);
</span></code></pre>
<ol>
<li>We're defining a pointer to a function, named <strong>greet_ptr</strong></li>
<li>We're using the <strong>*</strong> notation to signify that it is a pointer.</li>
<li>The function pointed to, should take one argument of type char pointer <strong>(char *)</strong>.</li>
<li>The function pointed to, should return nothing: <strong>void</strong>.</li>
<li>We need parenthesis, otherwise it becomes <strong>void *greet_ptr(char *)</strong>, a function taking a string and returning a <strong>void pointer.</strong></li>
</ol>
<br>
<p>The next interesting lines are:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span>greet_ptr = </span><span style="color:#569cd6;">&</span><span>greet;
</span><span>greet_ptr(</span><span style="color:#d69d85;">"foo"</span><span>);
</span></code></pre>
<ol>
<li>greet_ptr still points to greet function.</li>
<li>let's execute the code living at the address stored by greet_ptr.</li>
<li>This executable, <code>greet</code> is waiting for a parameter of type <code>char *</code>.</li>
<li>So the parameter <code>"foo"</code> will be passed to the function <code>greet</code> before execution.</li>
</ol>
<br>
<h3 id="what-is-the-address-of-a-function"><strong>What is the address of a function?</strong></h3>
<p>Before going further, we have to keep one thing in mind. <strong>a function name (label) is the address of the function.</strong> Yes. A <code>function name (label)</code> is converted to a <code>pointer to itself</code>. This means that function names can be assigned to a function pointer without using <code>&</code> to pass an address. Example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include</span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span>void (</span><span style="color:#569cd6;">*</span><span>greet_ptr)(</span><span style="color:#569cd6;">char *</span><span>);
</span><span>void (</span><span style="color:#569cd6;">*</span><span>hello_ptr)(</span><span style="color:#569cd6;">char *</span><span>);
</span><span>
</span><span style="color:#569cd6;">void </span><span>greet(</span><span style="color:#569cd6;">char *</span><span>name){
</span><span> printf(</span><span style="color:#d69d85;">"Hello </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">!</span><span style="color:#e3bbab;">\n\n</span><span style="color:#d69d85;">"</span><span>, name);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main(){
</span><span> greet_ptr = </span><span style="color:#569cd6;">&</span><span>greet;
</span><span> hello_ptr = greet; </span><span style="color:#608b4e;">// notice, there is no &
</span><span>
</span><span> </span><span style="color:#569cd6;">if</span><span>(greet_ptr == hello_ptr)
</span><span> printf(</span><span style="color:#d69d85;">"Pointers containing the same address.</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span>
</span><span> greet_ptr(</span><span style="color:#d69d85;">"foo"</span><span>);
</span><span> hello_ptr(</span><span style="color:#d69d85;">"foo"</span><span>);
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span></code></pre>
<br>
<h3 id="function-pointers-as-arguments"><strong>Function pointers as arguments?</strong></h3>
<p>Remember, a function name (label) is converted to a pointer to itself, so it can be passed whenever we need a function pointer. So, we can write function taking function pointers as arguments. Example:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include</span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span style="color:#569cd6;">void </span><span>sayHello(</span><span style="color:#569cd6;">char *</span><span>name){
</span><span> printf(</span><span style="color:#d69d85;">"Hello </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">!</span><span style="color:#e3bbab;">\n\n</span><span style="color:#d69d85;">"</span><span>, name);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>sayHi(</span><span style="color:#569cd6;">char *</span><span>name){
</span><span> printf(</span><span style="color:#d69d85;">"Hey </span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">!</span><span style="color:#e3bbab;">\n\n</span><span style="color:#d69d85;">"</span><span>, name);
</span><span>}
</span><span>
</span><span style="color:#608b4e;">// a function that takes a function and a string as parameters
</span><span style="color:#569cd6;">void </span><span>greet(</span><span style="color:#569cd6;">void </span><span>(</span><span style="color:#569cd6;">*</span><span>fctPtr)(</span><span style="color:#569cd6;">char *</span><span>), </span><span style="color:#569cd6;">char *</span><span>name) {
</span><span> fctPtr(name);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>greetBar(){
</span><span> greet(sayHello, </span><span style="color:#d69d85;">"bar"</span><span>);
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main () {
</span><span> greet(sayHello, </span><span style="color:#d69d85;">"foo"</span><span>);
</span><span> greet(sayHi, </span><span style="color:#d69d85;">"baz"</span><span>);
</span><span> greetBar();
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span></code></pre>
<br>
<p>The first interesting lines are:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#569cd6;">void </span><span>greet(</span><span style="color:#569cd6;">void </span><span>(</span><span style="color:#569cd6;">*</span><span>fctPtr)(</span><span style="color:#569cd6;">char *</span><span>), </span><span style="color:#569cd6;">char *</span><span>name) {
</span><span> fctPtr(name);
</span><span>}
</span></code></pre>
<ol>
<li>We're defining a function, named <code>greet</code>.</li>
<li>The <code>greet</code> function takes 2 arguments:
<ol>
<li>a function pointer: <code>void (*fctPtr)(char *)</code></li>
<li>a char pointer: <code>char *name</code></li>
</ol>
</li>
<li>The function pointed to, should be called with an argument of type <code>char *</code></li>
<li>The function pointed to is then called: <code>fctPtr(name)</code></li>
</ol>
<br>
<p>The next interesting line is:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span>greet(sayHello, </span><span style="color:#d69d85;">"foo"</span><span>);
</span></code></pre>
<ol>
<li><code>sayHello</code> is a function name (label)
<ol>
<li>that will be converted to a pointer to itself.</li>
<li>then, that pointer will be passed to <code>greet</code> function.</li>
</ol>
</li>
<li>The function <code>sayHello</code> will be called with <code>"foo"</code> as parameter.</li>
</ol>
<br>
<h3 id="we-can-also-use-function-pointers-as-return-values"><strong>We can also use function pointers as return values.</strong></h3>
<p>We <em>already</em> know a pointer can be returned from a function. A function pointer, <em>is</em> a <em>pointer</em>, so there is no big deal here.</p>
<br>
<h3 id="the-original-python-example-in-c"><strong>The original Python example in C?</strong></h3>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include</span><span style="color:#d69d85;"><stdio.h>
</span><span>
</span><span style="color:#569cd6;">int </span><span>square (</span><span style="color:#569cd6;">int </span><span>x) {
</span><span> </span><span style="color:#569cd6;">return</span><span> x </span><span style="color:#569cd6;">*</span><span> x;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>cube (</span><span style="color:#569cd6;">int </span><span>x) {
</span><span> </span><span style="color:#569cd6;">return</span><span> x </span><span style="color:#569cd6;">*</span><span> x </span><span style="color:#569cd6;">*</span><span> x;
</span><span>}
</span><span>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main () {
</span><span> </span><span style="color:#569cd6;">int</span><span> choice, value;
</span><span> </span><span style="color:#608b4e;">// an array of pointers to functions
</span><span> </span><span style="color:#608b4e;">// those functions should take an int and return an int
</span><span> </span><span style="color:#569cd6;">int </span><span>(</span><span style="color:#569cd6;">*</span><span>fp[</span><span style="color:#b5cea8;">2</span><span>])(</span><span style="color:#569cd6;">int</span><span>) = {square, cube};
</span><span> printf(</span><span style="color:#d69d85;">"Square (1) or Cube (2) ?: "</span><span>);
</span><span> scanf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">&</span><span>choice);
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"Value ?: "</span><span>);
</span><span> scanf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d</span><span style="color:#d69d85;">"</span><span>, </span><span style="color:#569cd6;">&</span><span>value);
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">*</span><span>fp[choice-</span><span style="color:#b5cea8;">1</span><span>])(value));
</span><span>
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span></code></pre>
<br>
<h3 id="any-real-life-examples"><strong>Any real life examples?</strong></h3>
<p>Function pointers gives C programmers, <strong>first-class citizen</strong> <a href="https://en.wikipedia.org/wiki/First-class_function">functionality</a>: being able to pass functions as argument to other functions, and being able to return functions as value from other functions.</p>
<br>
<h3 id="more-on-the-topic"><strong>More on the topic:</strong></h3>
<ul>
<li><a href="http://stackoverflow.com/questions/10758811/c-syntax-for-functions-returning-function-pointers/10759352#10759352">C syntax for functions returning function pointers</a></li>
</ul>
My journey to type checking 7521 lines of Python2021-11-08T03:40:17+00:002021-11-08T03:40:17+00:00
Unknown
https://nskm.xyz/posts/stcmp2/<p>I've spent the last couple of months adding <a href="http://mypy-lang.org/">type hints</a> to the <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=nsukami">RQL package</a>. In the <a href="https://nskm.xyz/posts/stcmp/">first</a> part, I talked you <strong>the reasons</strong> I think Mypy may be useful. In this part, I'll talk about the <strong>process I've followed</strong> for adding type annotations to RQL.</p>
<h2 id="recap-of-the-first-part">Recap of the first part:</h2>
<blockquote>
<p>Type hints are mainly useful for the <a href="https://hynek.me/talks/python-foss/">maintainers</a>, <strong><em>the human readers</em></strong> of <strong><em>large code bases</em></strong>. The people who <strong>read</strong> and <strong>debug</strong> and <strong>maintain</strong> large software systems <strong>they did not write</strong>.</p>
</blockquote>
<blockquote>
<p>The major benefit for you won't be fewer bugs or unit tests, but the ability to more easily read and reason about code written by someone else.</p>
</blockquote>
<br>
<h2 id="outline">Outline:</h2>
<p>First, I'll talk about what <a href="/posts/stcmp2/#mypy">Mypy</a> is and what should be it's initial <a href="/posts/stcmp2/#initial-setup">setup</a>.</p>
<p>Secondly, I'll briefly show you how I deconstructed the project (<a href="/posts/stcmp2/#import-structure">import structure</a> & <a href="/posts/stcmp2/#classes-hierarchy">classes hierarchy</a>) to have another perspective of how the project is built.</p>
<p>I will also show 3 different ways to <a href="/posts/stcmp2/#type-hints-generation">generate</a> type annotations and <a href="/posts/stcmp2/#type-information-inline-or-via-stubs">where</a> I think those type annotations should reside.</p>
<p>Finaly, I'll talk about things that still need to be <a href="/posts/stcmp2/#improvements">improved</a>. Mainly, the use of <code>Any</code> and <code>type:ignore</code>.</p>
<p>Feel free to jump straight to the part that's most interesting.</p>
<br>
<h2 id="mypy">Mypy:</h2>
<p><a href="https://mypy.readthedocs.io/en/stable/existing_code.html#using-mypy-with-an-existing-codebase">Mypy</a> is a library that provides optional static type checking. Unlike other programming languages, where the static type-checking takes place at compilation time, Mypy CLI does the type-check on-demand. At Logilab, the type-check was done, locally when developing, but also in the Continuous Integration pipeline, using Gitlab and Tox.</p>
<br>
<h2 id="initial-setup">Initial setup:</h2>
<p>We need to ensure that Mypy is really being run everywhere it should be and everytime someone pushes code. We don't want to spend time applying type annotations that won't be checked. If you don't check type annotations on a regular basis, they may become wrong after a certain period of time. That means:</p>
<h3 id="1-configuring-mypy">1. Configuring Mypy</h3>
<p>Configuration is done using a <code>mypy.ini</code> <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/blob/532de9f4788887a4c5ff315ad9a515033979c822/mypy.ini">file</a>. At the begining, we do not want strict typing checks. Then, we'll try to work towards more strigency, for example: <a href="https://mypy.readthedocs.io/en/stable/command_line.html#disallow-dynamic-typing">dissallowing dynamic typing</a> or <a href="https://mypy.readthedocs.io/en/stable/command_line.html#untyped-definitions-and-calls">disallowing untyped definitions</a>. We're working with a legacy code base with no type definitions, let's start with more relaxed options.</p>
<h3 id="2-ignoring-external-libraries">2. Ignoring external libraries:</h3>
<p>I've done it many weeks later, but I could have done it right from the beginning: tell Mypy to <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/commit/66c3196a4d2925183b6f6c943f6368943c3e4522">ignore</a> missing type hints from <a href="https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-type-hints-for-third-party-library">external libraries</a>. Maybe in another phase, we'll annotate them, but for now, they should be ignored.</p>
<h3 id="3-set-a-tox-environment-and-ci-job">3. Set a Tox environment and CI job:</h3>
<p>Add Tox environment to automate the boring stuff and reduce the <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/blob/branch/default/tox.ini#L44">boilerplate</a>. And add a Gitlab CI <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/blob/branch/default/.gitlab-ci.yml#L39">job</a>. that will be in charge of running Mypy every time a new changeset is sent to the Gitlab server.</p>
<br>
<h2 id="import-structure">Import structure</h2>
<p>I thought a diagram showing the import structure would help me understand the project. I also thought making a graph of the RQL's import structure would help me add types a bit faster. My idea was to add annotations from bottom to top. I was wrong. More than once, after adding type annotations to a module, I was obliged to go back to where it was already done. For example, method <a href="https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides">overrides</a>.</p>
<p><a href="https://github.com/thebjorn/pydeps">PyDeps</a> is the module dependency visualization I used, the result being:</p>
<p><img src="/images/import_structure2.svg" alt="" title="Right click & Open in new tab for better view" />
<br></p>
<p>If you open the diagram in another tab, you'll notice that we can see external libraries too, for example: <a href="https://forge.extranet.logilab.fr/open-source/logilab-database">logilab-database</a>, <a href="https://github.com/smurfix/yapps">yapps</a>, <a href="https://github.com/pygments/pygments">pygments</a>. Those are the ones that need to be ignored during type checking.</p>
<br>
<h2 id="classes-hierachy">Classes hierachy:</h2>
<p>Within <a href="https://pylint.org/">Pylint</a>, there is a tool named <a href="https://www.logilab.org/blogentry/6883">Pyreverse</a> that analyses Python code and extracts UML class diagrams and package dependencies. Using Pyreverse, we can have a clear view of all the classes and the relation between them:</p>
<p><img src="/images/classes_RQL2.svg" alt="" title="Right click & Open in new tab for better view" /></p>
<br>
<h2 id="type-hints-addition">Type hints addition:</h2>
<p>There are different options for adding type annotations.</p>
<h3 id="1-manual">1. Manual</h3>
<p>We can do it manually: finding what are the types flowing through our program, <em>I've been there, it's painful</em>. Literally:</p>
<ul>
<li>Adding breakpoints here and there</li>
<li>Running the function or running the tests</li>
<li>Once inside PDB, asking the <code>type</code> function to tell me who is who</li>
</ul>
<br>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">73</span><span>]: </span><span style="color:#ff3333;">def </span><span>g(x):
</span><span> </span><span style="color:#569cd6;">...</span><span>: breakpoint()
</span><span> </span><span style="color:#569cd6;">...</span><span>: x[</span><span style="color:#b5cea8;">5</span><span>] = </span><span style="color:#d69d85;">'foo'
</span><span> </span><span style="color:#569cd6;">...</span><span>: y = </span><span style="color:#b5cea8;">42
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#608b4e;"># a long list of non understandable code
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>y
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">76</span><span>]: </span><span style="color:#608b4e;"># let's assume we launched the tests and somewhere, g is called
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">77</span><span>]: g([</span><span style="color:#b5cea8;">16</span><span>, </span><span style="color:#b5cea8;">72</span><span>, </span><span style="color:#b5cea8;">38</span><span>, </span><span style="color:#b5cea8;">45</span><span>, </span><span style="color:#b5cea8;">21</span><span>, </span><span style="color:#b5cea8;">34</span><span>])
</span><span>> <ipython-input-</span><span style="color:#b5cea8;">75</span><span>-ef8444a3ab59>(</span><span style="color:#b5cea8;">3</span><span>)g()
</span><span>-> x[</span><span style="color:#b5cea8;">5</span><span>] = </span><span style="color:#d69d85;">'foo'
</span><span>(Pdb) type(x) </span><span style="color:#608b4e;"># please, what is the type of the parameter x?
</span><span><</span><span style="color:#ff3333;">class </span><span style="color:#d69d85;">'list'</span><span>>
</span><span>(Pdb) n
</span><span>> <ipython-input-</span><span style="color:#b5cea8;">75</span><span>-ef8444a3ab59>(</span><span style="color:#b5cea8;">4</span><span>)g()
</span><span>-> y = </span><span style="color:#b5cea8;">42
</span><span>(Pdb) n
</span><span>> <ipython-input-</span><span style="color:#b5cea8;">75</span><span>-ef8444a3ab59>(</span><span style="color:#b5cea8;">6</span><span>)g()
</span><span>-> </span><span style="color:#569cd6;">return </span><span>y
</span><span>(Pdb) type(y) </span><span style="color:#608b4e;"># please, what is the type of the returned value?
</span><span><</span><span style="color:#ff3333;">class </span><span style="color:#d69d85;">'int'</span><span>>
</span><span>(Pdb)
</span></code></pre>
<br>
<p>After that first run, we may be temped to write our function like one that takes a list and returns an integer:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>In [</span><span style="color:#b5cea8;">80</span><span>]: </span><span style="color:#608b4e;"># g takes a list and return an integer
</span><span>
</span><span>In [</span><span style="color:#b5cea8;">81</span><span>]: </span><span style="color:#ff3333;">def </span><span>g(x: list) -> int:
</span><span> </span><span style="color:#569cd6;">...</span><span>: x[</span><span style="color:#b5cea8;">5</span><span>] = </span><span style="color:#d69d85;">'foo'
</span><span> </span><span style="color:#569cd6;">...</span><span>: y = </span><span style="color:#b5cea8;">42
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#608b4e;"># a long list of non understandable code
</span><span> </span><span style="color:#569cd6;">...</span><span>: </span><span style="color:#569cd6;">return </span><span>y
</span><span> </span><span style="color:#569cd6;">...</span><span>:
</span></code></pre>
<br>
<p>I said <em>"temped"</em> because, maybe the function takes a dictionary and returns an integer. Only an extensive set of tests will tell us what's the best answer.<br />
<br></p>
<h3 id="2-reveal">2. Reveal:</h3>
<p>I never really use them, but another option is to use <code>reveal_type</code> or <code>reveal_locals</code> <a href="https://mypy.readthedocs.io/en/latest/common_issues.html#displaying-the-type-of-an-expression">functions</a>. You just put a <code>reveal_type(expr)</code> in the code, and run it with Mypy. I never really use those 2 functions because most of the time, the revealed type is <code>Any</code>:</p>
<p><img src="/images/reveal.png" alt="" /><br />
<br></p>
<h3 id="3-pyannotate">3. PyAnnotate:</h3>
<p>The third <em>and really helpful</em> option I want to suggest is, using a tool that will give you an <strong>exhaustive list of suggestions</strong>, for example PyAnnotate. <a href="https://github.com/dropbox/pyannotate">PyAnnotate</a> will help you list all the <strong>call arguments and all the return types observed at runtime</strong>.</p>
<p>Similar projects exists, <a href="https://github.com/Instagram/MonkeyType">MonkeyType</a> created by Instagram. <a href="https://google.github.io/pytype/">Pytype</a> from Google. And <a href="https://pyre-check.org/">Pyre</a> created at Facebook... <em>Hem, I mean, Meta</em>. I've used PyAnnotate just because it felt like the most easiest to setup and to use. I should probably give a try to the other options.</p>
<p>In the repository, there is a configuration <a href="https://github.com/dropbox/pyannotate/blob/master/example/example_conftest.py">example</a> you can use with Pytest. And after running your tests, you should find a new file named <code>type_info.json</code>. This file will contain a json that looks like:</p>
<pre data-lang="json" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-json "><code class="language-json" data-lang="json"><span> {
</span><span> </span><span style="color:#d69d85;">"path"</span><span>: </span><span style="color:#d69d85;">"rql/__init__.py"</span><span>,
</span><span> </span><span style="color:#d69d85;">"line"</span><span>: </span><span style="color:#b5cea8;">128</span><span>,
</span><span> </span><span style="color:#d69d85;">"func_name"</span><span>: </span><span style="color:#d69d85;">"RQLHelper.compute_solutions"</span><span>,
</span><span> </span><span style="color:#d69d85;">"type_comments"</span><span>: [
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Union, None, None, int) -> Set"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Insert, None, None, int) -> Set"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Union, Dict[str, function], None, int) -> Set"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Delete, None, None, int) -> Set"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Set, None, None, int) -> Set"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Union, Dict[str, function], Dict[str, str], int) -> Set[str]"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Union, None, None, int) -> pyannotate_runtime.collect_types.NoReturnType"</span><span>,
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Insert, None, None, int) -> pyannotate_runtime.collect_types.NoReturnType"
</span><span> ],
</span><span> </span><span style="color:#d69d85;">"samples"</span><span>: </span><span style="color:#b5cea8;">65
</span><span> },
</span><span> {
</span><span> </span><span style="color:#d69d85;">"path"</span><span>: </span><span style="color:#d69d85;">"rql/__init__.py"</span><span>,
</span><span> </span><span style="color:#d69d85;">"line"</span><span>: </span><span style="color:#b5cea8;">152</span><span>,
</span><span> </span><span style="color:#d69d85;">"func_name"</span><span>: </span><span style="color:#d69d85;">"RQLHelper.simplify"</span><span>,
</span><span> </span><span style="color:#d69d85;">"type_comments"</span><span>: [
</span><span> </span><span style="color:#d69d85;">"(rql.stmts.Union) -> None"
</span><span> ],
</span><span> </span><span style="color:#d69d85;">"samples"</span><span>: </span><span style="color:#b5cea8;">14
</span><span> },
</span></code></pre>
<p>Those generated type-hints are <strong>suggestions</strong>. You will have to tweak them first, then apply them. Once applied, you should run mypy on the updated files to verify that everything stays green.</p>
<br>
<h2 id="how-much-should-be-typed">How much should be typed?</h2>
<p>I won't deny, I was <a href="https://mypy.readthedocs.io/en/stable/existing_code.html#using-mypy-with-an-existing-codebase">gradually</a> adding annotation to everything, everywhere. There was a feeling of endlessness.</p>
<p><img src="/images/everywhere.gif" alt="" /></p>
<p>Then I asked myself: <em>"How do I know the RQL package has enough annotations? How do I know that I've done enough?"</em></p>
<p>I decided to <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/commit/68f034d32bb10d8ce8c7244c361afe219cc0cca8">follow</a> a path: adding Mypy <a href="https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-txt-report">coverage reports</a> to the project. The lower the percentage, the happier you should be:</p>
<pre data-lang="txt" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-txt "><code class="language-txt" data-lang="txt"><span>Mypy Type Check Coverage Summary
</span><span>================================
</span><span>
</span><span>Script: index
</span><span>
</span><span>+---------------------+-------------------+----------+
</span><span>| Module | Imprecision | Lines |
</span><span>+---------------------+-------------------+----------+
</span><span>| rql | 29.63% imprecise | 324 LOC |
</span><span>| rql._exceptions | 0.00% imprecise | 48 LOC |
</span><span>| rql.analyze | 62.35% imprecise | 603 LOC |
</span><span>| rql.base | 13.19% imprecise | 288 LOC |
</span><span>| rql.interfaces | 11.54% imprecise | 78 LOC |
</span><span>| rql.nodes | 24.20% imprecise | 1438 LOC |
</span><span>| rql.parser | 77.19% imprecise | 1245 LOC |
</span><span>| rql.parser.__main__ | 19.30% imprecise | 57 LOC |
</span><span>| rql.pygments_ext | 23.21% imprecise | 56 LOC |
</span><span>| rql.rqlgen | 1.62% imprecise | 247 LOC |
</span><span>| rql.rqltypes | 12.92% imprecise | 178 LOC |
</span><span>| rql.stcheck | 49.25% imprecise | 867 LOC |
</span><span>| rql.stmts | 30.98% imprecise | 1372 LOC |
</span><span>| rql.undo | 27.39% imprecise | 387 LOC |
</span><span>| rql.utils | 20.72% imprecise | 333 LOC |
</span><span>+---------------------+-------------------+----------+
</span><span>| Total | 38.64% imprecise | 7521 LOC |
</span><span>+---------------------+-------------------+----------+
</span></code></pre>
<br>
<p>Mypy does <a href="https://mypy.readthedocs.io/en/stable/type_inference_and_annotations.html#type-inference">type inference</a>, it can guess the types of values based on the context. We see in the picture below that even though there is no type annotations, the program is not flagged as 100% imprecise:
<img src="/images/mypy_reports_one.png" alt="" /><br />
<br></p>
<p>I finally <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/commit/00efe84a7e18f71cf4741dbd71736b6549dfbd0b">gave up</a>, to focus on what other <em>Logilabians</em>, suggested: <a href="https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/snippets/61#LC41">find</a> and <a href="https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/218">list</a> what are the <strong>RQL's most imported/used classes and functions</strong>. If those classes and functions are typed, we'll consider we've reach a milestone.<br />
<br></p>
<h2 id="type-information-inline-or-via-stubs">Type information inline or via stubs?</h2>
<p>Mypy gives you the ability to add type annotations to your project without ever modifying the original source code. You do it using what's called "stub files". <a href="https://mypy.readthedocs.io/en/stable/stubs.html">Stub</a> files are python-like files, that only contain type-checked variable, function, and class definitions. Nothing else.</p>
<p>I went the other way, I've added type annotations directly within the original source code. In the beginning, it really felt like the easiest thing to do, having everything in the same file. But now that there is more and more types, I think I'm better improve readability.</p>
<p>I can for example write <a href="https://mypy.readthedocs.io/en/stable/kinds_of_types.html#type-aliases">type aliases</a> whenever <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/blob/branch/default/rql/stcheck.py#L158">necessary</a>. Interesting inspiration can be found <a href="https://github.com/dropbox/incubator-superset-internal/blob/4cf3f99ee881a10995212828494f0b126dd5415e/superset/typing.py">here</a>. I can also write a snippet that will strip all those annotations and put them inside <code>.pyi</code> files.</p>
<p>From now on, I think I'll always use stub files.</p>
<br>
<h2 id="type-checking">Type Checking:</h2>
<p>The typing module defines a <a href="https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING">TYPE_CHECKING</a> constant that is <code>False</code> at runtime but treated as <code>True</code> while type checking. This technique is used to tell mypy something without the code being evaluated at runtime, for example resolving import cycles. Example <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/blob/branch/default/rql/nodes.py#L51">here</a>, another one <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/blob/branch/default/rql/stmts.py#L243">here</a>.</p>
<br>
<h2 id="improvements">Improvements:</h2>
<h3 id="1-less-any"><em>1. Less Any:</em></h3>
<p>I've been into situations where I'm not sure what should be the correct type annotation. Instead of leaving an empty type annotation, I put <code>Any</code>, to show other developers that the lack of a restrictive type hint is a conscious choice: either I don't know, either I'm not sure.</p>
<p>Normally, I should use <a href="https://docs.python.org/3/library/typing.html#the-any-type">Any</a> only if it's impossible for me to know what the type is going to be.</p>
<p>From the documentation we can <a href="https://docs.python.org/3/library/typing.html#the-any-type">read</a>:</p>
<blockquote>
<p>Notice that no type checking is performed when assigning a value of type Any to a more precise type. For example, the static type checker did not report an error when assigning a to s even though s was declared to be of type str and receives an int value at runtime!</p>
</blockquote>
<p>So, to take advantage of Mypy, I should really avoid using <a href="https://forge.extranet.logilab.fr/search?group_id=23&nav_source=navbar&page=1&project_id=300&repository_ref=branch%2Fdefault&scope=&search=%3A+Any&search_code=true&snippets=false">Any</a> and see if I can replace it with a proper annotation.</p>
<h3 id="2-less-type-ignore-comments"><em>2. Less type: ignore comments:</em></h3>
<p>It was also impossible to progress without using <code># type: ignore</code> comments, to work around the trickier cases. There are still too many <a href="https://forge.extranet.logilab.fr/search?group_id=23&nav_source=navbar&page=1&project_id=300&search=%3A+ignore">type: ignore</a> comments. I should have another look and see if I can improve the type annotations and remove some of those type: ignore comments.</p>
<br>
<h2 id="thanks">Thanks:</h2>
<p>Special thanks to <a href="https://www.linkedin.com/in/romaricst">Romaric</a> who took the time to read this article and provide insightful critiques.</p>
<br>
<h2 id="more-on-this-topic">More on this topic:</h2>
<p>Really hope you've learned something reading this article. As always, the topic is wide and cannot be completely covered here. To discover more about this subject, I strongly recommend the following links:</p>
<ul>
<li><a href="https://github.com/typeddjango/awesome-python-typing">Awesome typing</a></li>
<li><a href="https://yewtu.be/watch?v=8wjntHrTGPs">Liskov Substitution Principle with Johan Vergeer</a></li>
<li><a href="https://dropbox.tech/application/our-journey-to-type-checking-4-million-lines-of-python">Our journey to type checking 4 million lines of Python</a></li>
<li><a href="https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c">Static analysis at scale: an Instagram story</a></li>
</ul>
To type or not to type? That is the question2021-08-22T03:40:17+00:002021-08-22T03:40:17+00:00
Unknown
https://nskm.xyz/posts/stcmp/<p>I've spent the last couple of months adding <a href="http://mypy-lang.org/">type hints</a> to the <a href="https://forge.extranet.logilab.fr/cubicweb/RQL/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=nsukami">RQL package</a>. In this first part, I'll try to tell you <strong>when</strong> I think Mypy may be useful. In a <a href="https://nskm.xyz/posts/stcmp2/">second part</a>, I'll talk about the <strong>how</strong>, the process I've followed for adding type annotations to RQL. In a last part, I'll talk about what I've <strong>learned</strong> during the process.</p>
<br>
<h2 id="what">What?</h2>
<p>Yes, in case you were living under a rock, here are the news: you can now write Python <strong>as if</strong> it was a static programming language. The official <a href="https://www.python.org/dev/peps/pep-0484/">PEP</a>, the official <a href="http://www.mypy-lang.org/">website</a>. The official <a href="https://mypy.readthedocs.io/en/stable/">documentation</a>. And the official <a href="https://github.com/python/mypy">repository</a> <sup class="footnote-reference"><a href="#4">1</a></sup>. The <a href="http://www.alorelang.org/">Alore</a> programming language, the project Mypy borrows heavily from.</p>
<br>
<h2 id="why">Why?</h2>
<p>I remember when I was asked to work on this task. The (my) skepticism was real. Around me, reactions were mixed (confused, disgusted, interesed). I won't deny, sometimes, I'm really wondering if I'm not better go and learn to write <a href="https://ocaml.org/learn/">OCaml</a>. Python devs are so used to quickly prototype their programs, without worrying about the types of their variables, without worrying about their return types, etc. You cannot just come and tell them that type hints are nice! They need to be convinced.</p>
<p><img src="/images/what.gif" alt="" /></p>
<p>More than once, the discussion ended with the "explicit vs implicit", "static vs dynamic" holy wars, etc... I'm not trying to fuel those debates. There are pros and cons on each sides. You have to know what are the tools, what suits you best, depending on what you're trying to achieve, depending on what is your position (user, author, maintainer).</p>
<p>Assuming you're a person who already write Python programs, and you want to discover more about Mypy. If you only look at some introductory examples, they will help you understand <strong>what's going on.</strong> They won't <strong>convince</strong> you to jump on the type-hints band wagon. The problem with most of the examples is, they're too short, or too basic.</p>
<p>No disrepect to the authors, I guess, the intent was to make things easier to understand. Short and simple to get to the point. But short and simple would barely convince you to try.</p>
<p>So, what are the situations in which adding types to your program will be useful, necessary, not a waste of time, priceless, meaningful? To this question, I would say this:</p>
<blockquote>
<p>Type hints are mainly useful for the <a href="https://hynek.me/talks/python-foss/">maintainers</a>, <strong><em>the human readers</em></strong> of <strong><em>large code bases</em></strong>. The people who <strong>read</strong> and <strong>debug</strong> and <strong>maintain</strong> large software systems <strong>they did not write</strong>.</p>
</blockquote>
<blockquote>
<p>The major benefit for you won't be fewer bugs or unit tests, but the ability to more easily read and reason about code written by someone else.</p>
</blockquote>
<br>
<h2 id="i-need-to-know-the-types">I need to know the types:</h2>
<p>When you read/debug/maintain a large software system you've not written, you'll often be in the position in which you ask yourself: "Those function arguments, those variables, what are their types?", or "Is this function returning something?".</p>
<p>Remember, you were reading the source code, now you need to: modifiy the source code, add a couple of: <code>print(foo)</code>, <code>type(bar)</code>, <code>isinstance(baz)</code>, <code>qux.__class__</code>, <code>corge.__dict__</code>. You will probably add <a href="https://docs.python.org/3/library/functions.html#breakpoint">breakpoints</a> here and there, to dive inside <a href="https://docs.python.org/3/library/pdb.html">pdb</a>. Then start a REPL, then launch the program, then do some instrospection, then see what's going on. And finally, discover that the type is not what you were expecting.</p>
<p><img src="https://c.tenor.com/vZrArZhxAUoAAAAd/hugh-laurie-facepalm.gif" alt="phew" /></p>
<br>
<h2 id="what-about-just-reading-the-type">What about just reading the type?</h2>
<p>Let's be honest, as a <em>reader</em>, as the one who review millions of lines of code, you just want to be able to <em>read</em> what are the types. And I'm not talking about <a href="https://www.python.org/dev/peps/pep-0257/">docstrings</a>. Everybody should document their programs and write docstrings. But let's face it, docstrings may be missing, misleading, out of date, not clear enough, etc.... Let's compare the 2 following <em>shorts</em> snippets:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span> </span><span style="color:#569cd6;">def </span><span>get_description(self, tr): </span><span style="color:#608b4e;"># What is tr ???
</span><span> foo = self.get_foo()
</span><span> </span><span style="color:#569cd6;">if </span><span>foo != </span><span style="color:#d69d85;">"Bar"</span><span>:
</span><span> </span><span style="color:#569cd6;">return </span><span>tr(foo) </span><span style="color:#608b4e;"># ah, tr is a function. What's returned by tr?
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#d69d85;">"Bar" </span><span style="color:#608b4e;"># The method seems to return a string
</span></code></pre>
<p>Imagine having to find answers to those questions many times in a day?</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span> </span><span style="color:#569cd6;">def </span><span>get_description(self, tr: rt.TranslationFunction) -> Optional[str]:
</span><span> foo = self.get_foo()
</span><span> </span><span style="color:#569cd6;">if </span><span>foo != </span><span style="color:#d69d85;">"Bar"</span><span>:
</span><span> </span><span style="color:#569cd6;">return </span><span>tr(foo)
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#d69d85;">"Bar"
</span></code></pre>
<p>You see that in the second snippet, all the important informations are on the function header. You don't even read the function body and you already know that: <strong>"get_description is a method that takes a function as its only parameter and return a string or nothing"</strong>. That's a game changer when you're a maintainer.</p>
<p>It's not a surprise, <a href="https://google.github.io/pytype/">all</a> <a href="https://pyre-check.org/">the</a> <a href="https://github.com/Microsoft/pyright">big</a> <a href="https://monkeytype.readthedocs.io/en/latest/">companies</a>, the ones with large software systems, are already using type annotations.</p>
<br>
<h2 id="too-verbose-and-confusing-to-read">Too verbose and confusing to read?</h2>
<p>Adding type annotation can quickly lead to a source code that is too verbose and disgusting to read. The following is something I've written for RQL. I think I can do better (define type aliases, use stub files, use dataclasses, refactor, ...)</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">class </span><span>ScopeNode(</span><span style="color:#4ec9b0;">BaseNode</span><span>):
</span><span> </span><span style="color:#569cd6;">def </span><span>__init__(self):
</span><span> self.defined_vars: Dict[str, </span><span style="color:#d69d85;">"rql.nodes.Variable"</span><span>] = {}
</span><span> self.with_: List[</span><span style="color:#d69d85;">"rql.nodes.SubQuery"</span><span>] = []
</span><span> self.solutions: rt.SolutionsList = []
</span><span> self._varmaker = </span><span style="color:#569cd6;">None
</span><span> self.where: Optional[</span><span style="color:#d69d85;">"rql.base.Node"</span><span>] = </span><span style="color:#569cd6;">None
</span><span> self.having: Iterable[</span><span style="color:#d69d85;">"rql.base.Node"</span><span>] = ()
</span><span> self.schema: Optional[Any] = </span><span style="color:#569cd6;">None
</span><span> self.aliases: Dict[str, </span><span style="color:#d69d85;">"rql.nodes.ColumnAlias"</span><span>] = {}
</span></code></pre>
<br>
<h2 id="stub-files">Stub files?</h2>
<p>If you want to keep your source code free of type annotations, but still want to use type annotations, you can write <a href="https://www.python.org/dev/peps/pep-0484/#stub-files">stub files</a>. A stub file is just a file that will contain all your variables, function signatures, <em>annotated</em>. <strong>Function bodies in stub files are just a single ellipsis.</strong> Another really short example:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span> </span><span style="color:#608b4e;"># foo.py
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>get_description(self, tr):
</span><span> foo = self.get_foo()
</span><span> </span><span style="color:#569cd6;">if </span><span>foo != </span><span style="color:#d69d85;">"Bar"</span><span>:
</span><span> </span><span style="color:#569cd6;">return </span><span>tr(foo)
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#d69d85;">"Bar"
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span> </span><span style="color:#608b4e;"># foo.pyi <--- notice the extension used for stub files
</span><span>
</span><span> </span><span style="color:#569cd6;">def </span><span>get_description(self, tr: rt.TranslationFunction) -> Optional[str]: </span><span style="color:#569cd6;">...
</span></code></pre>
<br>
<h2 id="i-want-type-annotations-what-s-next">I want type annotations, what's next?</h2>
<p><a href="https://mypy.readthedocs.io/en/stable/introduction.html#introduction">Mypy</a> is a static type checker for Python. Let's see what Mypy can to for us.</p>
<br>
<h3 id="1-generating-type-annotations">1. Generating type annotations:</h3>
<p>Let's suppose you want to add type annotations to your code base. How do you proceed?</p>
<p>Mypy includes the <a href="https://mypy.readthedocs.io/en/stable/stubgen.html">stubgen</a> tool that can automatically generate stub files. For example, <a href="https://github.com/LeMeteore/typeshed/blob/b2e1083560e223ec415fcb4dd155366e2984b2a5/stubs/dateparser/dateparser/__init__.pyi">here</a> is an example of a <code>.pyi</code> file generated by the stubgen tool for the <a href="https://github.com/scrapinghub/dateparser">dateparser</a> package.</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">from </span><span>typing </span><span style="color:#9b9b9b;">import </span><span>Any, Optional
</span><span>
</span><span style="color:#569cd6;">def </span><span>parse(
</span><span> date_string: Any,
</span><span> date_formats: Optional[Any] = </span><span style="color:#569cd6;">...</span><span>,
</span><span> languages: Optional[Any] = </span><span style="color:#569cd6;">...</span><span>,
</span><span> locales: Optional[Any] = </span><span style="color:#569cd6;">...</span><span>,
</span><span> region: Optional[Any] = </span><span style="color:#569cd6;">...</span><span>,
</span><span> settings: Optional[Any] = </span><span style="color:#569cd6;">...</span><span>): </span><span style="color:#569cd6;">...
</span></code></pre>
<p>Keep in mind, most types annotations will default to <code>Any</code>. You'll need to update <code>pyi</code> files and write more precise types (We agree that <code>Any</code> everywhere is not really useful). <a href="https://github.com/hoefling/typeshed/blob/855acb4da416375b0d4791be51bf0934b0ee9e29/stubs/dateparser/dateparser/__init__.pyi">Here</a> is an example of the same stub file with more precise type annotations:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>datetime
</span><span style="color:#9b9b9b;">import </span><span>sys
</span><span style="color:#9b9b9b;">from </span><span>typing </span><span style="color:#9b9b9b;">import </span><span>Set, Tuple
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>dateparser.date </span><span style="color:#9b9b9b;">import </span><span>DateDataParser
</span><span>
</span><span style="color:#569cd6;">if </span><span>sys.version_info >= (</span><span style="color:#b5cea8;">3</span><span>, </span><span style="color:#b5cea8;">8</span><span>):
</span><span> </span><span style="color:#9b9b9b;">from </span><span>typing </span><span style="color:#9b9b9b;">import </span><span>Literal, TypedDict
</span><span style="color:#569cd6;">else</span><span>:
</span><span> </span><span style="color:#9b9b9b;">from </span><span>typing_extensions </span><span style="color:#9b9b9b;">import </span><span>Literal, TypedDict
</span><span>
</span><span>__version__: str
</span><span>
</span><span>_default_parser: DateDataParser
</span><span>
</span><span>_Part = Literal[</span><span style="color:#d69d85;">"day"</span><span>, </span><span style="color:#d69d85;">"month"</span><span>, </span><span style="color:#d69d85;">"year"</span><span>]
</span><span>_ParserKind = Literal[</span><span style="color:#d69d85;">"timestamp"</span><span>, </span><span style="color:#d69d85;">"relative-time"</span><span>, </span><span style="color:#d69d85;">"custom-formats"</span><span>, </span><span style="color:#d69d85;">"absolute-time"</span><span>, </span><span style="color:#d69d85;">"no-spaces-time"</span><span>]
</span><span>
</span><span style="color:#569cd6;">class </span><span>_Settings(</span><span style="color:#4ec9b0;">TypedDict</span><span>, total=</span><span style="color:#4ec9b0;">False</span><span>):
</span><span> DATE_ORDER: str
</span><span> PREFER_LOCALE_DATE_ORDER: bool
</span><span> TIMEZONE: str
</span><span> TO_TIMEZONE: str
</span><span> RETURN_AS_TIMEZONE_AWARE: bool
</span><span> PREFER_DAY_OF_MONTH: Literal[</span><span style="color:#d69d85;">"current"</span><span>, </span><span style="color:#d69d85;">"first"</span><span>, </span><span style="color:#d69d85;">"last"</span><span>]
</span><span> PREFER_DATES_FROM: Literal[</span><span style="color:#d69d85;">"current_period"</span><span>, </span><span style="color:#d69d85;">"future"</span><span>, </span><span style="color:#d69d85;">"past"</span><span>]
</span><span> RELATIVE_BASE: datetime.datetime
</span><span> STRICT_PARSING: bool
</span><span> REQUIRE_PARTS: list[_Part]
</span><span> SKIP_TOKENS: list[str]
</span><span> NORMALIZE: bool
</span><span> RETURN_TIME_AS_PERIOD: bool
</span><span> PARSERS: list[_ParserKind]
</span><span>
</span><span style="color:#569cd6;">def </span><span>parse(
</span><span> date_string: str,
</span><span> date_formats: list[str] | Tuple[str] | Set[str] | </span><span style="color:#569cd6;">None </span><span>= </span><span style="color:#569cd6;">...</span><span>,
</span><span> languages: list[str] | Tuple[str] | Set[str] | </span><span style="color:#569cd6;">None </span><span>= </span><span style="color:#569cd6;">...</span><span>,
</span><span> locales: list[str] | Tuple[str] | Set[str] | </span><span style="color:#569cd6;">None </span><span>= </span><span style="color:#569cd6;">...</span><span>,
</span><span> region: str | </span><span style="color:#569cd6;">None </span><span>= </span><span style="color:#569cd6;">...</span><span>,
</span><span> settings: _Settings | </span><span style="color:#569cd6;">None </span><span>= </span><span style="color:#569cd6;">...</span><span>,
</span><span>) -> datetime.datetime | </span><span style="color:#569cd6;">None</span><span>: </span><span style="color:#569cd6;">...
</span><span>
</span></code></pre>
<p>Do we agree that the :</p>
<ul>
<li>parse function parameters are more precise.</li>
<li>settings parameter is annotated with a more detailed type.<sup class="footnote-reference"><a href="#6">2</a></sup></li>
<li>returned value is more precise.</li>
<li>refactoring of this function will be easier.</li>
</ul>
<br>
<h3 id="2-type-checking">2. Type checking:</h3>
<p>Before running your tests, before running your program, Mypy can detect <em>some types</em> of bugs:</p>
<p><img src="/images/what2.png" alt="" /></p>
<p>I pretend <code>foo</code> should return a <code>string</code>, yet, there are cases where foo does return <code>None</code>. That should be fixed. Notice how, the program is running fine. Mypy will type check the program and tell you what's wrong. Mypy won't stop you from running the program.</p>
<br>
<h2 id="conclusion">Conclusion:</h2>
<p>Most people, including myself, don't work on projects of this size (<a href="https://dropbox.tech/application/our-journey-to-type-checking-4-million-lines-of-python">4 million lines of Python code</a>), and hence have not to deal with the constraints of such a big code base.</p>
<p>Most people can perfectly write their Python programs without using type annotations. Mypy is just another tool inside your toolbox. I just want you to try, that will certainly make you a better software developer.</p>
<p>In the <a href="https://nskm.xyz/posts/stcmp2/">next article</a>, I will talk a little bit more about the actual process I've followed for adding type annotations to the RQL package.</p>
<p><br><br></p>
<h2 id="to-learn-more-about-this-topic">To learn more about this topic:</h2>
<ul>
<li><a href="https://docs.python.org/3/library/typing.html">Python typing</a></li>
<li><a href="https://tube.incognet.io/watch?v=pJOmlFf5Je4">What Python can learn from Haskell</a></li>
<li><a href="https://www.youtube.com/watch?v=hTrjTAPnA_k">Type hinting (and mypy)</a></li>
</ul>
<div class="footnote-definition" id="4"><sup class="footnote-definition-label">1</sup>
<p>Which, by the way, is where you'll find more interesting answers than what's inside the documentation, imho.</p>
</div>
<div class="footnote-definition" id="6"><sup class="footnote-definition-label">2</sup>
<p>Yes. I won't deny, there is suddenly more lines to <strong>read</strong></p>
</div>
Elephant in the room (part 1)2021-05-25T03:40:17+00:002021-05-25T03:40:17+00:00
Unknown
https://nskm.xyz/posts/eitr/<p>I'm living in a Continent where (<em>it sounds like</em>) nobody really cares about software freedom, data privacy, surveillance. I guess, people here are simply too busy trying to stay <a href="https://www.africanews.com/2021/05/29/second-volcano-erupts-in-eastern-democratic-republic-of-congo/">alive</a>, not enough time to think about FOSS. Or maybe they're simply confused, maybe they simply don't know.</p>
<p>The topic is wide, it's easy to digress, and I'm not sure about the best way to tackle it (<em>consider this to be a first draft, updates will come</em>). Let's say it's just an attempt, an effort, to <em>at least</em> address the issue.</p>
<br>
Okay. Let me put you in the context...
<br>
<h1 id="will-you-be-okay-with-anybody-trying-to"><em>Will you be okay with anybody trying to:</em></h1>
<ul>
<li>Fix any of your furnitures.</li>
<li>Take you anywhere you want.</li>
<li>Offer you any food you want.</li>
<li>Pay for all your phone, sms, mail bills.</li>
<li>Let you watch any movie, any film, any show.</li>
<li>Alert you about something every 5mn.</li>
</ul>
<br>
<h1 id="and-in-exchange-the-same-person"><em>And in exchange the same person:</em></h1>
<p>Won't ask you for any money, nothing, he'll just keep a tiny tiny list of all:</p>
<ul>
<li>the furnitures in your house?</li>
<li>the places you visit?</li>
<li>the foods you eat?</li>
<li>the conversations you had?</li>
<li>the movies you watched?</li>
<li>the time he notified you? The reason he notified you?</li>
</ul>
<br>
<h1 id="do-we-agree-that-you-won-t-be-okay"><em>Do we agree that you won't be okay?</em></h1>
<p><em>In standard conditions for temperature and pressure</em>, you won't be okay with anybody offering you the deal above because (<em>among many other reasons</em>):</p>
<ul>
<li>you don't want anybody to access your <strong>private space</strong></li>
<li>you want to keep a certain level of <strong>autonomy</strong>, <strong>independance</strong>.</li>
<li>you want to preserve your physical <strong>integrity</strong>.</li>
</ul>
<br>
<h1 id="interestingly"><em>Interestingly:</em></h1>
<p>It sounds like, nobody want me to:</p>
<ul>
<li>put a webcam and a microphone in their house.</li>
<li>have the key to their house.</li>
<li>follow them.</li>
<li>take picture of them or recording them.</li>
</ul>
<p>Yet, most of the same people are using all kinds of <em>"privacy invading"</em> technologies on a daily basis: <a href="https://www.forbes.com/sites/kateoflahertyuk/2019/07/28/apple-siri-eavesdropping-puts-millions-of-users-at-risk">Siri</a>, <a href="https://www.cnet.com/home/smart-home/alexa-and-google-voice-assistants-app-exploits-left-it-vulnerable-to-eavesdropping/">Alexa</a>, <a href="https://www.vrt.be/vrtnws/en/2019/07/10/google-employees-are-eavesdropping-even-in-flemish-living-rooms/">Google</a>. I won't even talk about <a href="https://www.dw.com/en/samsung-admits-smart-tv-may-be-eavesdropping/a-18246372">smart TVs</a>, nor <a href="https://www.reuters.com/article/us-eu-facebook-dataprotection-idUSKCN1UO1B4">Facebook</a>.</p>
<br>
<h1 id="any-reasons-that-let-us-give-our-datas-to-the-tech-giants"><em>Any reasons that let us give our datas to the Tech Giants?</em></h1>
<p>People use privacy invading software for maaaaany reasons that can be understood.
<br></p>
<p>There is the "I need to figure out what to eat today" reason. Present all over the world, especially for the poorer people. Even more true in the Black continent.</p>
<p>There is the fact that <strong>implications</strong> (of using privacy invading software) <strong>are invisible</strong>, not obvious. While the benefits are immediate, concrete, observable.</p>
<p>Terms of Service are, <em>most of the time</em>, <a href="https://tosdr.org/en/service/182">confusing</a>, <a href="https://tosdr.org/fr/service/217">difficult</a>, <a href="https://tosdr.org/fr/service/198">blurry</a>.</p>
<p>There is also the "I use it because people around me are using it" reason. People are moved by <a href="https://en.wikipedia.org/wiki/Network_effect">those around</a> them and wish to fit in as quickly as possible, as simply as possible.</p>
<br>
<p><a href="https://en.wikipedia.org/wiki/Henrik_Ibsen">Henrik Ibsen</a> said <em>"A thousand words leave not the same deep impression as does a single deed"</em>. And I definitely agree with him:</p>
<center>
<img src="https://nskm.xyz/processed_images/vicious-cycle.42e644fb1aedfddb.png" />
</center>
<p><a href="https://theconversation.com/heres-how-tech-giants-profit-from-invading-our-privacy-and-how-we-can-start-taking-it-back-120078">Source</a></p>
<br>
<h1 id="do-you-know-that"><em>Do you know that:</em></h1>
<ol>
<li>Your right to physical integrity was <em>already</em> affirmed in 1222, <a href="https://en.wikipedia.org/wiki/Kouroukan_Fouga#Contents">inside the Manden Charter</a>, article 5:</li>
</ol>
<blockquote>
<p>Everybody has a right to life and to the <strong>preservation of physical integrity</strong>. [...]</p>
</blockquote>
<ol start="2">
<li>Your right to privacy was affirmed in 1948 by the <a href="https://www.un.org/en/about-us/universal-declaration-of-human-rights">Universal Declaration of Human Rights</a>, article 12:</li>
</ol>
<blockquote>
<p>No one shall be subjected to <strong>arbitrary interference</strong> with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.</p>
</blockquote>
<ol start="3">
<li>Within the <a href="https://www.echr.coe.int/documents/convention_eng.pdf">European Convention on Human Rights</a>, in the article 8, we can read:</li>
</ol>
<blockquote>
<p>Right to respect for private and family life.</p>
<ol>
<li>
<p>Everyone has the right to respect for his private and family life, his home and his correspondence.</p>
</li>
<li>
<p>There shall be <strong>no interference by a public authority</strong> with the exercise of this right except such as is in accordance with the law and is necessary in a democratic society in the interests of national security, public safety or the economic well-being of the country, for the prevention of disorder or crime, for the protection of health or morals, or for the protection of the rights and freedoms of others.</p>
</li>
</ol>
</blockquote>
<ol start="4">
<li>The <a href="https://www.law.cornell.edu/constitution">U.S. constitution</a> <a href="https://constitution.congress.gov/constitution/amendment-4/">Fourth Amendment</a> states that:</li>
</ol>
<blockquote>
<p>The right of the people to be secure in their persons, houses, papers, and effects, <strong>against unreasonable searches and seizures</strong>, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.</p>
</blockquote>
<br>
<p>The point seems to be: <strong>Every human being has a right to privacy, and this right should not be violated.</strong></p>
<h1 id="what-is-privacy"><em>What is privacy?</em></h1>
<p>I was not able to find a fixed, simple, perfect, static definition of <a href="https://www.nolo.com/dictionary/right-to-privacy-term.html">privacy</a>. The definition may vary from one country to another, from one constitution to another, from one period of time to another.</p>
<p>But I guess we can agree that privacy is any information that intrudes into the intimacy of the individual. <em>Right</em>? Privacy is that thing which is essential to the protection of our dignity as Human Being. <em>Right</em>? Anything that is directly related to :</p>
<ul>
<li>your personal space, your house, your image,</li>
<li>your financial status, your medical status,</li>
<li>your religious, political, <em>or even sexual</em> orientations,</li>
</ul>
<p><strong><em>Privacy is the fundamental way to protect yourself</em></strong></p>
<h1 id="some-of-the-solutions-we-can-explore"><em>Some of the solutions we can explore:</em></h1>
<p>How do we defend ourselves and our friends from privacy invading technologies? What can we do to safeguard our personal privacy. What are the ways we can use to better protect ourselves when going online? What are the alternatives to the privacy invading software? I'll talk about the solutions in the next <a href="https://nskm.xyz/posts/wmdop/">article</a> (<em>coming soon</em>).</p>
<br>
<h1 id="no-i-m-not"><em>No, I'm not:</em></h1>
<ul>
<li>Saying that I've all the answers.</li>
<li>Saying that you should do as I say.</li>
<li>Asking you to close all your accounts.</li>
<li>Asking you to go back to the stone age.</li>
<li>Trying to make you affraid of technology.</li>
<li>Saying that I've found all the alternatives.</li>
<li>Saying that migrating from privacy invading software will be an easy task.</li>
</ul>
<br>
<h1 id="what-i-m-asking-you"><em>What I'm asking you:</em></h1>
<p>Completely shuting down all the Tech Giants won't be an easy task. It cannot be done within the snap of a finger <em>(I wish it <a href="https://yewtu.be/watch?v=TWB31WFomz4">could</a>)</em>. Let's stay optimistic, there is still some actionable steps we can take to help:</p>
<ul>
<li>Ask yourself :
<ul>
<li>Why are you using those platforms?</li>
<li>What are the services you really need?</li>
<li>Is there any alternatives? (Respecting your privacy).</li>
</ul>
</li>
<li>Be aware:
<ul>
<li>of the risks, of the pros & cons.</li>
<li>that if it's free, you're probably the <a href="https://ethannonsequitur.com/wp-content/uploads/2011/09/facebook-and-you-pigs.jpg">product</a></li>
</ul>
</li>
<li>Ask for more:
<ul>
<li>Transparency to all those platforms.</li>
<li>Regulation & protection to your leaders, your governments.</li>
</ul>
</li>
<li><strong>Spark conversation, everywhere.</strong></li>
</ul>
<br>
<h1 id="interesting-links">Interesting links:</h1>
<p>This topic is too wide to be covered entirely in this tiny article. I really hope you're learned something. I strongly recommend the following links for a better exploration and understanding of the topic.</p>
<p>The following is a list of resources you should <em>download, browse & share</em>:</p>
<ul>
<li><a href="#">Enemy of the State</a></li>
<li><a href="https://www.killswitchthefilm.com/">Killswitch</a></li>
<li><a href="https://yewtu.be/watch?v=ZHl0WI32XkY">We are legion</a></li>
<li><a href="#">Terms and conditions may apply</a></li>
<li><a href="https://citizenfourfilm.com/">Citizenfour</a></li>
<li><a href="https://www.thesocialdilemma.com/">Social dilemma</a></li>
<li><a href="https://www.thegreathack.com/">The great hack</a></li>
<li><a href="https://snowdenfilm.com/">Snowden</a></li>
<li><a href="https://www.yewtu.be/watch?v=YXmMtbgdces">Nos données personnelles valent de l'or</a></li>
<li><a href="https://activism.net/cypherpunk/manifesto.html">A Cypherpunk's Manifesto</a></li>
<li><a href="https://www.yewtu.be/watch?v=IWzmYE60_dc">Macron trahit la « start-up nation », les gafam raflent la mise</a></li>
</ul>
<p><br><br></p>
<h1 id="thanks">Thanks:</h1>
<p>Special <a href="https://www.meme-arsenal.com/memes/379eb2a9776a3bbaac72286e3d857cbf.jpg">thanks</a> to Orbifx, <a href="https://mastodon.scot/@orbifx">the Nature's emissary</a>, and <a href="https://mastodon.social/@travis_kalanick">Travis Kalanick</a>. They both gave me interesting ideas to help me finish this article.</p>
<br>
<h1 id="do-not-forget">Do not forget:</h1>
<center>
<img src="https://nskm.xyz/processed_images/real-power-is-people.815a6f16ec1b60fc.jpg" />
</center>
Django single file project. Again2020-07-10T01:09:16+00:002020-07-10T01:09:16+00:00
Unknown
https://nskm.xyz/posts/dsfpa/<p>This article is a follow up on the <a href="https://nskm.xyz/posts/dsfp/">one</a> I've written few years
ago. Special shoutout and thanks and hugs and kudos to <a href="http://buzzi.me/">Javier
Buzzi</a> for taking the time to <a href="https://gist.github.com/kingbuzzman/d7859d9734b590e52fad787d19c34b52">improve</a> the original script.</p>
<p>TL;DR: <a href="https://bin.nskm.xyz/pastes/ee784daff75aae62">download old version</a> & <a href="https://bin.nskm.xyz/pastes/c4434741122bec0f">download new version</a></p>
<br />
<h2 id="very-important-disclaimer">VERY IMPORTANT DISCLAIMER</h2>
<p>The content associated with mlvin.xyz may be very offensive to some people. I've lost the ownership of mlvin.xyz a very long time ago, sad situation. I'm not anymore linked to this domain name, nor to the content hosted on this website. <em>By the way F*** OVH</em>.</p>
<h2 id="what-are-those-improvements">What are those improvements?</h2>
<h3 id="application-label">Application label?</h3>
<p>Our models are not defined as in a classical Django application. So,
each model should declare which app it belongs to. You can read more
about app_label Meta option <a href="https://docs.djangoproject.com/en/3.0/ref/models/options/#django.db.models.Options.app%5Flabel">here</a>.</p>
<p>The problem with the first script was duplication:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">class </span><span>Author(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> name = models.CharField(max_length=</span><span style="color:#b5cea8;">200</span><span>)
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> app_label = APP_LABEL
</span><span>
</span><span style="color:#569cd6;">class </span><span>Book(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> author = models.ForeignKey(Author, related_name=</span><span style="color:#d69d85;">'books'</span><span>)
</span><span> title = models.CharField(max_length=</span><span style="color:#b5cea8;">400</span><span>)
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> app_label = APP_LABEL
</span></code></pre>
<p>We want to define app_label once and for all our models:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># The current name of the file, which will be the name of our app
</span><span>APP_LABEL, _ = os.path.splitext(os.path.basename(os.path.abspath(__file__)))
</span></code></pre>
<p>Then, we need to overwrite the Django function in charge of:</p>
<ul>
<li>looking for an application configuration</li>
<li>attach the model to the found/correct application configuration.</li>
</ul>
<p>This function is named <a href="https://github.com/django/django/blob/stable/3.0.x/django/apps/registry.py#L243">get_containing_app_config</a></p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">def </span><span>get_containing_app_config(module):
</span><span> </span><span style="color:#569cd6;">if </span><span>module == </span><span style="color:#d69d85;">'__main__'</span><span>:
</span><span> </span><span style="color:#608b4e;"># if inside single file project,
</span><span> </span><span style="color:#608b4e;"># retrieve the application configuration using APP_LABEL variable
</span><span> </span><span style="color:#569cd6;">return </span><span>apps.get_app_config(APP_LABEL)
</span><span>
</span><span> </span><span style="color:#608b4e;"># otherwise, do as you always do
</span><span> </span><span style="color:#569cd6;">return </span><span>apps._get_containing_app_config(module)
</span><span>
</span><span style="color:#608b4e;"># save the original function
</span><span>apps._get_containing_app_config = apps.get_containing_app_config
</span><span>
</span><span style="color:#608b4e;"># ask Django to use our custom function
</span><span>apps.get_containing_app_config = get_containing_app_config
</span></code></pre>
<h3 id="migrations-folder">Migrations folder?</h3>
<p>There is a Django settings named <a href="https://github.com/django/django/blob/stable/3.0.x/django/conf/global_settings.py#L616">MIGRATION_MODULE</a>. According to the <a href="https://github.com/django/django/blob/stable/3.0.x/docs/ref/settings.txt#L2021">documentation</a>:</p>
<blockquote>
<p>A dictionary specifying the package where migration modules can be found on a per-app basis. The default value of this setting is an empty dictionary, but the default package name for migration modules is ``migrations``.</p>
</blockquote>
<br>
<p>In the first version of the script, all the migrations were put in a folder named <code>migrations</code>. With the following configuration, we can:</p>
<ul>
<li>name the migration folder how we want.</li>
<li>make sure that we have one migration folder per application.</li>
</ul>
<br>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># Where migrations folder need to be created
</span><span>APP_MIGRATION_MODULE = </span><span style="color:#d69d85;">'</span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">_migrations' </span><span>% APP_LABEL
</span><span>APP_MIGRATION_PATH = os.path.join(BASE_DIR, APP_MIGRATION_MODULE)
</span><span>
</span><span style="color:#608b4e;"># Create the migration folder
</span><span style="color:#608b4e;"># and a __init__.py file if necessary:
</span><span style="color:#569cd6;">if not </span><span>os.path.exists(APP_MIGRATION_PATH):
</span><span> os.makedirs(APP_MIGRATION_PATH)
</span><span> open(os.path.join(APP_MIGRATION_PATH, </span><span style="color:#d69d85;">'__init__.py'</span><span>), </span><span style="color:#d69d85;">'w'</span><span>).close()
</span></code></pre>
<p>We also took the time to update the Django settings, somewhere inside the call to <code>settings.configure function</code>, you'll find:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># Django needs to be told where is the migration module
</span><span>MIGRATION_MODULES={APP_LABEL: APP_MIGRATION_MODULE},
</span></code></pre>
<h3 id="fake-modules">Not so fake modules?</h3>
<p>A correctly configured Django application should normally have a <code>model modules</code>, a <code>urls modules</code> and a <code>tests module</code>. Python provides us with an interesting <a href="https://docs.python.org/3/library/types.html#types.ModuleType">class</a>: <code>types.ModuleType</code>, that will help us create a module, <em>you guessed it</em>:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;"># Used so you can do 'from <name of file>.models import *'
</span><span>models_module = ModuleType(</span><span style="color:#d69d85;">'</span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">.models' </span><span>% (APP_LABEL))
</span><span>tests_module = ModuleType(</span><span style="color:#d69d85;">'</span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">.tests' </span><span>% (APP_LABEL))
</span><span>urls_module = ModuleType(</span><span style="color:#d69d85;">'</span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">.urls' </span><span>% (APP_LABEL))
</span><span>
</span><span style="color:#608b4e;"># we created the urlpatterns list earlier,
</span><span style="color:#608b4e;"># let's attach this variable to the urls_module:
</span><span>urls_module.urlpatterns = urlpatterns
</span><span>
</span><span style="color:#608b4e;"># A model module needs to have classes that inherits from models.Model
</span><span style="color:#608b4e;"># let's correctly add all our models we defined earlier
</span><span style="color:#608b4e;"># as attributes to the models_module:
</span><span style="color:#569cd6;">for </span><span>variable_name, value </span><span style="color:#569cd6;">in </span><span>list(locals().items()):
</span><span> </span><span style="color:#608b4e;"># We are only interested in models
</span><span> </span><span style="color:#569cd6;">if </span><span>inspect.isclass(value) </span><span style="color:#569cd6;">and </span><span>issubclass(value, models.Model):
</span><span> setattr(models_module, variable_name, value)
</span><span>
</span><span style="color:#608b4e;"># Setup the fake modules
</span><span>sys.modules[models_module.__name__] = models_module
</span><span>sys.modules[urls_module.__name__] = urls_module
</span><span>sys.modules[APP_LABEL].models = models_module
</span><span>sys.modules[APP_LABEL].urls = urls_module
</span></code></pre>
<h3 id="going-further">Going further?</h3>
<p>I really hope you've learned something, I did \o/. You may want to browse some of the gists kept by <a href="https://gist.github.com/search?q=mlvin">Javier Buzzi</a>.
<br><br>
You may also be really interested by the work done for <a href="https://github.com/zenwalker/django-micro">django-micro</a> a lightweight wrapper around Django that turns it into a microframework.
<br><br>
If you have any kind of interesting improvements, advices, suggestions, please send me an email, or post a comment on <a href="https://mastodon.social/@lemeteore/104489090816400323">Mastodon</a>. <em>Still not took the time to install a comment system, shame on me</em>.</p>
<p>o/</p>
Free and Open Source Software2017-04-04T00:00:00+00:002017-04-04T00:00:00+00:00
Unknown
https://nskm.xyz/posts/foss/<p>I'm often asked why do I prefer to use free software in places (Dakar, Libreville, Kinshasa, ...), where basically every human being is using closed source and proprietary software? Here are some of the reasons.</p>
<br>
<h3 id="not-being-able-to-pay-for-a-license"><strong>Not being able to pay for a license</strong></h3>
<p>Learning informatics was equivalent to installing a bunch of software requiring me to pay for a license. And coming from a very modest family, paying for a license was not an option. I've never really been able to explain to my parent that</p>
<ul>
<li>I need to pay, via internet, using a credit card</li>
<li>a company located somewhere in the world, I don't even know where</li>
<li>to get the permission to use such an abstract thing like a software</li>
<li>for some time, then I'll have to pay again, to renew the permission</li>
</ul>
<br>
<h3 id="tired-of-looking-for-the-crack"><strong>Tired of looking for the crack</strong></h3>
<p>Not being able to pay for the license, the remaining choice was to look for the crack. that was the most easy way to get the software you want to use. But in the mean time, That was a very difficult task, always browsing the internet, looking for the product keys, the key generator, the patch, the full activator, etc... It was time consuming, and boring as editors are always trying to make it more and more difficult.</p>
<br>
<h3 id="trying-to-be-a-good-citizen"><strong>Trying to be a good citizen</strong></h3>
<p>Even if they should, most people often do not feel bad about using cracked software. After all, there were no physical harm, no shoplifting, no unlawful entry into a building, no indiscriminate taking of goods. Why feeling bad about it?</p>
<p>Theft, from <a href="https://en.wikipedia.org/wiki/Theft">Wikipedia</a></p>
<blockquote>
<p>In common usage, a theft is the illegal taking of another person's property or services without that person's permission or consent with the intent to deprive the rightful owner of it.</p>
</blockquote>
<p>Property, from <a href="https://en.wikipedia.org/wiki/Property#Types_of_property">Wikepedia</a></p>
<blockquote>
<p>Important widely recognized types of property include real property (the combination of land and any improvements to or on the land), personal property (physical possessions belonging to a person), private property (property owned by legal persons, business entities or individual natural persons), public property (state owned or publicly owned and available possessions) and intellectual property (exclusive rights over artistic creations, inventions, etc.), although the last is not always as widely recognized or enforced.</p>
</blockquote>
<br>
Above are the main reasons why I chose free and open source software. You can actually close this page, and go back to your awesome life, **thanks for reading**. But there is more.
<br>
<h3 id="the-free-and-open-source-software-philosophy-promotes-values-that-go-beyond-software"><strong>The free and open source software philosophy promotes values that go beyond software</strong></h3>
<ul>
<li>It's about the right to <em>own</em> an item, <em>to use it for any purpose</em></li>
<li>It's about the right to <em>study</em> how this item is made</li>
<li>It's about the right to <em>customize</em> this item</li>
<li>It's about the right to <em>share</em> the item modified or not, with whoever you want</li>
</ul>
<br>
<h3 id="fine-can-we-have-some-real-life-examples-please"><strong>Fine, can we have some real life examples please?</strong></h3>
<p>From <a href="https://www.softwareheritage.org/mission/heritage/">Software Heritage</a></p>
<blockquote>
<p>Software controls the electronic equipment embedded in the machines we use to travel, communicate, trade and exchange. Software is just crucial for the proper operation of every large organizations. Software has enabled the emergence of new forms of social and political organizations. Software is essential to understand our society.</p>
</blockquote>
<p>Maybe it's not perceptible enough, but free and open source software is already <a href="http://embedded-computing.com/articles/open-source-software-everywhere/">everywhere</a>. Here you are:</p>
<ul>
<li>Aerospace: Open Source Software projects from <a href="https://code.nasa.gov/">NASA</a></li>
<li>Automotive: <a href="https://www.automotivelinux.org">AGL</a> group, including lots of <a href="https://www.automotivelinux.org/about/members">automakers</a></li>
<li>Robotics: <a href="https://www.osrfoundation.org/osrf-projects/">Open Source Robotics Foundation</a></li>
<li>Healthcare: <a href="http://www.open-emr.org/">Open-EMR</a>, <a href="http://openmrs.org/">OpenMRS</a></li>
<li>Education: <a href="http://open-sankore.org/en">Open Sankoré</a>, an interactive whiteboard software.</li>
<li>Films: <a href="https://www.youtube.com/watch?v=eRsGyueVLvQ">Sintel</a>, and many other open movies by <a href="https://www.blender.org/features/projects/">Blender Foundation</a>.</li>
</ul>
<br>
It is no more about software, it is about **openess and freedom**.
<br>
The topic is actually very huge and deep, I've not even mentioned privacy, nor surveillance. I obviously won't be able to cover the whole thing.
<h3 id="for-those-who-want-to-go-further-i-strongly-recommands-the-following-links"><strong>For those who want to go further, I strongly recommands the following links:</strong></h3>
<ul>
<li><a href="https://www.youtube.com/watch?v=xKsR5jWl1Tk">Threats of data mining</a>.</li>
<li><a href="https://www.youtube.com/watch?v=8vOF8VsUwbA">Freedom in the cloud</a></li>
<li><a href="http://modernfarmer.com/2016/07/right-to-repair/">Right to fix their own dang tractors</a></li>
<li><a href="https://opensource.com/article/17/4/become-open-source-sustainer">Why you should become an open source sustainer</a></li>
<li><a href="https://anybox.fr/blog/pourquoi-l-open-source">Pourquoi l'open source</a> (French)</li>
</ul>
Threads2017-03-08T04:30:25+00:002017-03-08T03:40:17+00:00
Unknown
https://nskm.xyz/posts/an-overwiew-on-threads/<p>I've written something about <a href="https://nskm.xyz/posts/processes-introduction/">processes</a> Let's write some notes about threads. This is not really an introduction to threads. It's more like a little bit of introspection, so we can have an interesting perspective of what are threads.<br><br></p>
<h4 id="what-s-really-a-thread"><strong>What's really a thread?</strong></h4>
<p>Processes each have their own address space. Threads exist as subsets of a process. Threads are just multiple workers in the same <a href="https://nskm.xyz/posts/processes-introduction/">virtual address space</a> all threads in a process share the same memory. They can also share open files and other resources. Within that VAS, each thread has its own ID, its own stack, its own program counter, its own independent flow of control, its own registers set. A thread is just a <strong>context of execution</strong>.</p>
<br>
<p><img src="/images/threads-example.gif" alt="Threads" /></p>
<p><br><em>Image from <a href="http://mooneegee.blogspot.sn/2015/01/os-thread.html">mooneegee.blogspot.sn</a></em></p>
<p><br><br></p>
<p>Next is a c script spawning two <em>posix</em> threads:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#define _GNU_SOURCE
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><pthread.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><unistd.h> </span><span style="color:#608b4e;">// sleep
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><sys/syscall.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><sys/types.h>
</span><span>
</span><span style="color:#569cd6;">void * </span><span>say_hello(</span><span style="color:#569cd6;">void* </span><span>data){
</span><span> </span><span style="color:#569cd6;">char *</span><span>str = (</span><span style="color:#569cd6;">char*</span><span>)data;
</span><span> </span><span style="color:#569cd6;">int</span><span> howmany = </span><span style="color:#b5cea8;">4</span><span>;
</span><span> </span><span style="color:#569cd6;">int</span><span> r = (rand() % </span><span style="color:#b5cea8;">20</span><span>);
</span><span> pid_t thread_id = syscall(__NR_gettid);
</span><span>
</span><span> </span><span style="color:#569cd6;">while</span><span>(howmany > </span><span style="color:#b5cea8;">0</span><span>){
</span><span> sleep(r);
</span><span> howmany = howmany - </span><span style="color:#b5cea8;">1</span><span>;
</span><span>
</span><span> printf(</span><span style="color:#d69d85;">"</span><span style="color:#b4cea8;">%s</span><span style="color:#d69d85;">, "</span><span>,str);
</span><span> printf(</span><span style="color:#d69d85;">"from thread </span><span style="color:#b4cea8;">%u</span><span style="color:#d69d85;"> and process </span><span style="color:#b4cea8;">%u</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>,
</span><span> thread_id, getpid());
</span><span> }
</span><span> </span><span style="color:#569cd6;">return NULL</span><span>;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">int </span><span>main(</span><span style="color:#569cd6;">void</span><span>){
</span><span> pthread_t t1,t2;
</span><span>
</span><span> </span><span style="color:#569cd6;">if </span><span>(getpid() < </span><span style="color:#b5cea8;">0</span><span>) {
</span><span> perror(</span><span style="color:#d69d85;">"unable to get pid"</span><span>);
</span><span> } </span><span style="color:#569cd6;">else </span><span>{
</span><span> printf(</span><span style="color:#d69d85;">"The main process id is </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, getpid());
</span><span> }
</span><span>
</span><span> pthread_create(</span><span style="color:#569cd6;">&</span><span>t1,</span><span style="color:#569cd6;">NULL</span><span>,say_hello,</span><span style="color:#d69d85;">"howdy"</span><span>);
</span><span> pthread_create(</span><span style="color:#569cd6;">&</span><span>t2,</span><span style="color:#569cd6;">NULL</span><span>,say_hello,</span><span style="color:#d69d85;">"hello"</span><span>);
</span><span> pthread_join(t1,</span><span style="color:#569cd6;">NULL</span><span>);
</span><span> pthread_join(t2,</span><span style="color:#569cd6;">NULL</span><span>);
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span></code></pre>
<br>
<p>If we <a href="http://man7.org/linux/man-pages/man1/strace.1.html">strace</a> the execution of our c script:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>$ strace -f ./bin/how_to_thread
</span></code></pre>
<p>The output will display lots of informations. And for what we are doing, the most interesting lines will look like this:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>clone(child_stack=0x7f30fa9b8ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f30fa9b99d0, tls=0x7f30fa9b9700, child_tidptr=0x7f30fa9b99d0) = 7450
</span><span>Process 7450 attached
</span><span>[pid 7449] clone(child_stack=0x7f30fa1b7ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f30fa1b89d0, tls=0x7f30fa1b8700, child_tidptr=0x7f30fa1b89d0) = 7451
</span></code></pre>
<br>
<h4 id="the-clone-system-call"><strong>The clone() system call:</strong></h4>
<p><a href="https://nskm.xyz/posts/processes-introduction/">Processes</a> are created with the <a href="http://man7.org/linux/man-pages/man2/fork.2.html">fork()</a> system call. However, there is a separate system call, named <a href="http://man7.org/linux/man-pages/man2/clone.2.html">clone()</a> which is used for creating threads. It works like fork(), but it accepts a number of flags for adjusting its behavior so the child can share some parts of the parent's execution context. Our c script made a call to clone() twice. And looking at <em>some</em> of the flags that have been passed, we can see that:</p>
<p>the caller and the child process:</p>
<ul>
<li>CLONE_VM: run in the <strong>same memory space</strong>.</li>
<li>CLONE_FS: share the <strong>same filesystem information</strong>.</li>
<li>CLONE_FILES: share the <strong>same file descriptor table</strong>.</li>
<li>CLONE_THREAD: are placed in the <strong>same thread group</strong>.</li>
</ul>
<br>
<p>Another interesting arguments passed to clone system call are <code>child_stack</code> and <code>tls</code>. From the man page, we can read <a href="https://man7.org/linux/man-pages/man2/clone.2.html">somewhere</a>:</p>
<blockquote>
<p>The child_stack argument specifies the location of the stack used by the child process.</p>
</blockquote>
<p>And in <a href="http://man7.org/linux/man-pages/man2/set_thread_area.2.html">another part</a> of the man page:</p>
<blockquote>
<p>set a thread local storage (TLS) area</p>
</blockquote>
<p>Basically, we're allocating a <em>personal stack</em> to each of our threads. But, we also give each thread, an access to a region of memory which is <em>not shared</em> with all other threads. Like one may do with Python using the <a href="https://docs.python.org/3/library/threading.html#thread-local-data">threading.local</a> class.</p>
<p><br><br></p>
<h4 id="gdb-to-the-rescue"><strong>GDB to the rescue</strong></h4>
<p>Let's have a confirmation using <a href="https://www.gnu.org/software/gdb/">GDB</a>:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>$ gdb ./bin/how_to_thread
</span></code></pre>
<br>
<p>First, we set two breakpoints, one at the start of the main function, another one at the start of the say_hello function:</p>
<p><img src="/images/insert-breakpoints.png" alt="insert breakpoints" /></p>
<p><br><br></p>
<p>Then we start the debugged program: <code>r</code>, and list the mapped memory regions:</p>
<p><img src="/images/info-proc-map1.png" alt="info proc mappings" /></p>
<p><br><br></p>
<p>So far, we see only one stack segment, the one used by the main process, that is because our threads are not yet created. Let's continue a little bit (twice) with <code>c</code>, and you should be able to see something like this:</p>
<p><img src="/images/info-proc-map2.png" alt="info proc mappings" /></p>
<p><br><br></p>
<p>This time, we can see three stacks in the same virtual address space. The ones used by our threads are <code>[stack:15074]</code> and <code>[stack:15075]</code>. GDB to inspect each thread's stack:</p>
<p><img src="/images/stacks.png" alt="thread's stack" /></p>
<p><br><br></p>
<p>LWP is an acronym and stands for Light Weight Process. Next to LWP is the thread ID assigned by the kernel to this thread. We see that each thread has its own stack, and inside each stack, the progress is different, while thread 3 is entering the say_hello function, thread 2 is in <em>nanosleep</em>. By the way nanosleep is also a <a href="http://man7.org/linux/man-pages/man2/nanosleep.2.html">system call</a>.</p>
<p><br><br></p>
<p>Here we are, a thread is indeed just a <strong>context of execution</strong>.</p>
<p><br><br></p>
<p>I won't be able to cover the whole topic as it's not an easy and small one. So you may find <em>really</em> interesting the following links:</p>
<ul>
<li><a href="http://man7.org/linux/man-pages/man7/pthreads.7.html">Pthreads</a></li>
<li><a href="https://lwn.net/Articles/7577/">Making Linux safe for pthreads</a></li>
<li><a href="http://www.linfo.org/context_switch.html">Context switch</a></li>
<li><a href="https://0xax.gitbooks.io/linux-insides/content/SyncPrim/sync-3.html">Semaphores</a></li>
<li><a href="http://turnoff.us/geek/dont-share-mutable-state/">The real reason not to share a mutable state</a></li>
</ul>
Django single file project2017-01-12T17:17:17+00:002017-01-12T17:17:17+00:00
Unknown
https://nskm.xyz/posts/dsfp/<h4 id="disclaimer"><strong>Disclaimer?</strong></h4>
<p>The main purpose of this post is learning. Like any tool we should explore and find ways to bend it until it breaks. You should <em>normally</em> not use those techniques in real life projects <em>right</em>? For this post, I'm assuming you're using Django 1.10.4, and Python3.6.
<br><br></p>
<h4 id="single-file"><strong>Single file?</strong></h4>
<p>Before reading <a href="http://shop.oreilly.com/product/0636920032502.do">Lightweight Django</a>, I never tought it was possible to build a Django project in a single python file. For that kind of stuff, I usually think, <a href="http://flask.pocoo.org/">Flask</a>, or <a href="http://bottlepy.org/docs/dev/">Bottle</a>. And in fact, there are many <a href="http://codecondo.com/14-minimal-web-frameworks-for-python/">others</a>.
<br><br></p>
<p>After reading the first chapter, I was a little bit disappointed. There were no Django <a href="https://docs.djangoproject.com/en/1.10/topics/db/models/">models</a> involved in the shown example. So, I try to find out if it was possible: <strong>a single file Django project with at least one application containing one model</strong>. I found some interesting links on the topic. But either it was for a very old version of Django, either I was obliged to install another package besides Django. After few days reading others people code, this is what I've done:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;">#!/usr/bin/env python
</span><span>
</span><span style="color:#9b9b9b;">import </span><span>os
</span><span style="color:#9b9b9b;">import </span><span>sys
</span><span style="color:#9b9b9b;">from </span><span>django.conf </span><span style="color:#9b9b9b;">import </span><span>settings
</span><span>
</span><span>BASE_DIR = os.path.dirname(os.path.abspath(__file__))
</span><span>
</span><span>sys.path[</span><span style="color:#b5cea8;">0</span><span>] = os.path.dirname(BASE_DIR)
</span><span>
</span><span style="color:#608b4e;"># the current folder name will also be our app
</span><span>APP_LABEL = os.path.basename(BASE_DIR)
</span><span>
</span><span>
</span><span>settings.configure(
</span><span> DEBUG=os.environ.get(</span><span style="color:#d69d85;">'DEBUG'</span><span>, </span><span style="color:#d69d85;">'on'</span><span>) == </span><span style="color:#d69d85;">'on'</span><span>,
</span><span> SECRET_KEY=os.environ.get(</span><span style="color:#d69d85;">'SECRET_KEY'</span><span>, os.urandom(</span><span style="color:#b5cea8;">32</span><span>)),
</span><span> ALLOWED_HOSTS=os.environ.get(</span><span style="color:#d69d85;">'ALLOWED_HOSTS'</span><span>, </span><span style="color:#d69d85;">'localhost'</span><span>).split(</span><span style="color:#d69d85;">','</span><span>),
</span><span> ROOT_URLCONF=__name__,
</span><span> MIDDLEWARE=[
</span><span> </span><span style="color:#d69d85;">'django.middleware.security.SecurityMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.sessions.middleware.SessionMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.middleware.common.CommonMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.middleware.csrf.CsrfViewMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.auth.middleware.AuthenticationMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.messages.middleware.MessageMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.middleware.clickjacking.XFrameOptionsMiddleware'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.middleware.locale.LocaleMiddleware'</span><span>,
</span><span> ],
</span><span> INSTALLED_APPS=[
</span><span> APP_LABEL,
</span><span> </span><span style="color:#d69d85;">'django.contrib.admin'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.auth'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.contenttypes'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.sessions'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.messages'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.staticfiles'</span><span>,
</span><span> </span><span style="color:#d69d85;">'rest_framework'</span><span>,
</span><span> ],
</span><span> STATIC_URL=</span><span style="color:#d69d85;">'/static/'</span><span>,
</span><span> STATICFILES_DIRS=[
</span><span> os.path.join(BASE_DIR, </span><span style="color:#d69d85;">"static"</span><span>),
</span><span> ],
</span><span> STATIC_ROOT=os.path.join(BASE_DIR, </span><span style="color:#d69d85;">"static_root"</span><span>),
</span><span> MEDIA_ROOT=os.path.join(BASE_DIR, </span><span style="color:#d69d85;">"media"</span><span>),
</span><span> MEDIA_URL=</span><span style="color:#d69d85;">'/media/'</span><span>,
</span><span> TEMPLATES=[
</span><span> {
</span><span> </span><span style="color:#d69d85;">'BACKEND'</span><span>: </span><span style="color:#d69d85;">'django.template.backends.django.DjangoTemplates'</span><span>,
</span><span> </span><span style="color:#d69d85;">'DIRS'</span><span>: [os.path.join(BASE_DIR, </span><span style="color:#d69d85;">"templates"</span><span>),],
</span><span> </span><span style="color:#d69d85;">'APP_DIRS'</span><span>: </span><span style="color:#569cd6;">True</span><span>,
</span><span> </span><span style="color:#d69d85;">'OPTIONS'</span><span>: {
</span><span> </span><span style="color:#d69d85;">'context_processors'</span><span>: [
</span><span> </span><span style="color:#d69d85;">'django.template.context_processors.debug'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.template.context_processors.i18n'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.template.context_processors.request'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.auth.context_processors.auth'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.template.context_processors.tz'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.messages.context_processors.messages'</span><span>,
</span><span> ],
</span><span> },
</span><span> },
</span><span> ],
</span><span> DATABASES={
</span><span> </span><span style="color:#d69d85;">'default'</span><span>: {
</span><span> </span><span style="color:#d69d85;">'ENGINE'</span><span>: </span><span style="color:#d69d85;">'django.db.backends.sqlite3'</span><span>,
</span><span> </span><span style="color:#d69d85;">'NAME'</span><span>: os.path.join(BASE_DIR, </span><span style="color:#d69d85;">'db.sqlite3'</span><span>),
</span><span> }
</span><span> },
</span><span> REST_FRAMEWORK={
</span><span> </span><span style="color:#d69d85;">'DEFAULT_PERMISSION_CLASSES'</span><span>: [
</span><span> </span><span style="color:#d69d85;">'rest_framework.permissions.IsAdminUser'</span><span>,
</span><span> ],
</span><span> </span><span style="color:#d69d85;">'PAGE_SIZE'</span><span>: </span><span style="color:#b5cea8;">10
</span><span> }
</span><span>)
</span><span>
</span><span>
</span><span style="color:#9b9b9b;">import </span><span>django
</span><span>django.setup()
</span><span>
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>django.db </span><span style="color:#9b9b9b;">import </span><span>models
</span><span style="color:#9b9b9b;">from </span><span>django.contrib </span><span style="color:#9b9b9b;">import </span><span>admin
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>django.db </span><span style="color:#9b9b9b;">import </span><span>models
</span><span>
</span><span style="color:#608b4e;"># Create your models here.
</span><span>
</span><span style="color:#569cd6;">class </span><span>Author(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> name = models.CharField(max_length=</span><span style="color:#b5cea8;">200</span><span>)
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> app_label = APP_LABEL
</span><span>
</span><span style="color:#569cd6;">class </span><span>Book(</span><span style="color:#4ec9b0;">models.Model</span><span>):
</span><span> author = models.ForeignKey(Author, related_name=</span><span style="color:#d69d85;">'books'</span><span>)
</span><span> title = models.CharField(max_length=</span><span style="color:#b5cea8;">400</span><span>)
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> app_label = APP_LABEL
</span><span>
</span><span>admin.site.register(Book)
</span><span>admin.site.register(Author)
</span><span>admin.autodiscover()
</span><span>
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>serializers
</span><span>
</span><span style="color:#569cd6;">class </span><span>BookSerializer(</span><span style="color:#4ec9b0;">serializers.ModelSerializer</span><span>):
</span><span> </span><span style="color:#569cd6;">class </span><span>Meta:
</span><span> model = Book
</span><span> fields = </span><span style="color:#d69d85;">'__all__'
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>viewsets
</span><span>
</span><span style="color:#569cd6;">class </span><span>BooksViewSet(</span><span style="color:#4ec9b0;">viewsets.ReadOnlyModelViewSet</span><span>):
</span><span> queryset = Book.objects.all()
</span><span> serializer_class = BookSerializer
</span><span>
</span><span>
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>django.conf.urls </span><span style="color:#9b9b9b;">import </span><span>url, include
</span><span style="color:#9b9b9b;">from </span><span>rest_framework </span><span style="color:#9b9b9b;">import </span><span>routers
</span><span style="color:#9b9b9b;">from </span><span>django.http </span><span style="color:#9b9b9b;">import </span><span>HttpResponse
</span><span style="color:#9b9b9b;">from </span><span>django.contrib </span><span style="color:#9b9b9b;">import </span><span>admin
</span><span>
</span><span>router = routers.DefaultRouter()
</span><span>router.register(</span><span style="color:#569cd6;">r</span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">books</span><span style="color:#d69d85;">'</span><span>, BooksViewSet)
</span><span>
</span><span style="color:#569cd6;">def </span><span>index(request):
</span><span> </span><span style="color:#569cd6;">return </span><span>HttpResponse(</span><span style="color:#d69d85;">"Hello"</span><span>)
</span><span>
</span><span>
</span><span>urlpatterns = [
</span><span> url(</span><span style="color:#569cd6;">r</span><span style="color:#d69d85;">'</span><span style="color:#569cd6;">^</span><span style="color:#e3bbab;">admin/</span><span style="color:#d69d85;">'</span><span>, admin.site.urls),
</span><span> url(</span><span style="color:#569cd6;">r</span><span style="color:#d69d85;">'</span><span style="color:#569cd6;">^$</span><span style="color:#d69d85;">'</span><span>, index, name=</span><span style="color:#d69d85;">'homepage'</span><span>),
</span><span> url(</span><span style="color:#569cd6;">r</span><span style="color:#d69d85;">'</span><span style="color:#569cd6;">^</span><span style="color:#e3bbab;">api/</span><span style="color:#d69d85;">'</span><span>, include(router.urls)),
</span><span> url(</span><span style="color:#569cd6;">r</span><span style="color:#d69d85;">'</span><span style="color:#569cd6;">^</span><span style="color:#e3bbab;">api-auth/</span><span style="color:#d69d85;">'</span><span>, include(</span><span style="color:#d69d85;">'rest_framework.urls'</span><span>,\
</span><span> namespace=</span><span style="color:#d69d85;">'rest_framework'</span><span>))
</span><span>]
</span><span>
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>django.core.wsgi </span><span style="color:#9b9b9b;">import </span><span>get_wsgi_application
</span><span>
</span><span style="color:#569cd6;">if </span><span>__name__ == </span><span style="color:#d69d85;">"__main__"</span><span>:
</span><span> </span><span style="color:#9b9b9b;">from </span><span>django.core.management </span><span style="color:#9b9b9b;">import </span><span>execute_from_command_line
</span><span> execute_from_command_line(sys.argv)
</span><span style="color:#569cd6;">else</span><span>:
</span><span> get_wsgi_application()
</span></code></pre>
<p>Everything related to <a href="http://www.django-rest-framework.org/">Django REST Framework</a> is not mandatory. I was looking for <a href="https://xkcd.com/979/">interesting links</a> over the internet, and I found this question on <a href="http://stackoverflow.com/q/32866578/4055832">StackOverflow</a>. I saw this as an opportunity to kill two birds with one stone.
<br><br></p>
<p>The above script was saved in a file named <code>single.py</code> in a folder named <code>lwdj</code>. But you can do as you feel, as soon as you respect the name of the file and the name of the current folder. The rest is known story:</p>
<p>Make the migrations for our application:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span><span>⤷ python single.py makemigrations lwdj
</span><span>Migrations for 'lwdj':
</span><span> migrations/0001_initial.py:
</span><span> - Create model Author
</span><span> - Create model Book
</span><span>(dj) ~/.../how_to_django/lwdj (master *%) ⤷
</span></code></pre>
<br>
<p>Run the migrations:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span><span>⤷ python single.py migrate
</span><span>Operations to perform:
</span><span> Apply all migrations: admin, auth, contenttypes, lwdj, sessions
</span><span>Running migrations:
</span><span> Applying contenttypes.0001_initial... OK
</span><span> Applying auth.0001_initial... OK
</span><span> Applying admin.0001_initial... OK
</span><span> Applying admin.0002_logentry_remove_auto_add... OK
</span><span> Applying contenttypes.0002_remove_content_type_name... OK
</span><span> Applying auth.0002_alter_permission_name_max_length... OK
</span><span> Applying auth.0003_alter_user_email_max_length... OK
</span><span> Applying auth.0004_alter_user_username_opts... OK
</span><span> Applying auth.0005_alter_user_last_login_null... OK
</span><span> Applying auth.0006_require_contenttypes_0002... OK
</span><span> Applying auth.0007_alter_validators_add_error_messages... OK
</span><span> Applying auth.0008_alter_user_username_max_length... OK
</span><span> Applying lwdj.0001_initial... OK
</span><span> Applying sessions.0001_initial... OK
</span><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span></code></pre>
<br>
<p>Create a super user and run the server:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span><span>⤷ python single.py createsuperuser
</span><span>Username (leave blank to use 'nsukami'):
</span><span>Email address: nsukami@gmail.com
</span><span>Password:
</span><span>Password (again):
</span><span>Superuser created successfully.
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span><span>⤷ python single.py runserver
</span><span>Performing system checks...
</span><span>
</span><span>System check identified no issues (0 silenced).
</span><span>January 12, 2017 - 18:52:57
</span><span>Django version 1.10.4, using settings None
</span><span>Starting development server at http://127.0.0.1:8000/
</span><span>Quit the server with CONTROL-C.
</span></code></pre>
<p><br><br></p>
<h4 id="so-what-have-we-learned-here"><strong>So, what have we learned here?</strong></h4>
<p>Yes, that is not how you are supposed to use Django. But we were able to learn few things.
<br><br></p>
<h4 id="projects-and-applications"><strong>Projects and applications</strong></h4>
<p>Somewhere in the Django documentation related to <a href="https://docs.djangoproject.com/en/1.10/ref/applications/#projects-and-applications">projects and applications</a>, we can read:</p>
<blockquote>
<p>There's no restriction that a project package can't also be considered an application and have models, etc. (which would require adding it to INSTALLED_APPS).</p>
</blockquote>
<p>We can double check using the <code>tree</code> command:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span><span>⤷ tree
</span><span>.
</span><span>├── activate -> /home/nsukami/envs/dj/bin/activate
</span><span>├── db.sqlite3
</span><span>├── Makefile
</span><span>├── media
</span><span>├── migrations
</span><span>│ ├── 0001_initial.py
</span><span>│ ├── __init__.py
</span><span>│ └── __pycache__
</span><span>│ └── __init__.cpython-36.pyc
</span><span>├── single.py
</span><span>├── static
</span><span>│ └── site.css
</span><span>├── static_root
</span><span>└── templates
</span><span> └── home.html
</span></code></pre>
<br>
<h4 id="the-django-setup-method"><strong>The django.setup() method</strong></h4>
<p>Reading The Fabulous <a href="https://docs.djangoproject.com/en/dev/ref/applications/#how-applications-are-loaded">Manual</a>, we learn that, this method configures Django by loading the settings, but also:</p>
<blockquote>
<p>When Django starts, django.setup() is responsible for populating the application registry.</p>
</blockquote>
<p>Let's start a shell using the command <code>python single.py shell</code>, and let's check the application registry:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>(dj) >>> </span><span style="color:#9b9b9b;">from </span><span>django.apps </span><span style="color:#9b9b9b;">import </span><span>apps
</span><span>(dj) >>> apps.ready
</span><span style="color:#569cd6;">True
</span><span>(dj) >>>
</span><span>(dj) >>>
</span><span>(dj) >>> apps.get_app_configs()
</span><span>odict_values([<AppConfig: lwdj>, <AdminConfig: admin>, <AuthConfig: auth>, <ContentTypesConfig: contenttypes>, <SessionsConfig: sessions>, <MessagesConfig: messages>, <StaticFilesConfig: staticfiles>, <AppConfig: rest_framework>])
</span><span>(dj) >>>
</span></code></pre>
<br>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>(dj) >>> c = apps.get_app_config(</span><span style="color:#d69d85;">'lwdj'</span><span>)
</span><span>(dj) >>>
</span><span>(dj) >>> c.module
</span><span><module </span><span style="color:#d69d85;">'lwdj' </span><span>(namespace)>
</span><span>(dj) >>>
</span><span>(dj) >>> c.path
</span><span style="color:#d69d85;">'/home/nsukami/GITHUB/how_tos/how_to_django/lwdj'
</span><span>(dj) >>>
</span><span>(dj) >>> c.verbose_name
</span><span style="color:#d69d85;">'Lwdj'
</span><span>(dj) >>>
</span><span>(dj) >>> c.name
</span><span style="color:#d69d85;">'lwdj'
</span><span>(dj) >>> c.label
</span><span style="color:#d69d85;">'lwdj'
</span><span>(dj) >>>
</span><span>(dj) >>> </span><span style="color:#569cd6;">for </span><span>m </span><span style="color:#569cd6;">in </span><span>c.models: print(m)
</span><span style="color:#569cd6;">...
</span><span>author
</span><span>book
</span><span>(dj) >>>
</span></code></pre>
<p>Neat!
<br></p>
<p>We also <a href="https://docs.djangoproject.com/en/dev/ref/applications/#how-applications-are-loaded">read</a> that:</p>
<blockquote>
<p><em>At this stage, your code shouldn't import any models!</em></p>
</blockquote>
<p>This is exactly why our models, and everything related to them, do not appear before those 2 lines of code.</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#9b9b9b;">import </span><span>django
</span><span>django.setup()
</span></code></pre>
<br>
<p>Doing otherwise, raises the <code>AppRegistryNotReady</code> exception:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span><span>⤷ python single.py runserver
</span><span>Traceback (most recent call last):
</span><span> File "single.py", line 90, in <module>
</span><span> class Author(models.Model):
</span><span> File "/home/nsukami/envs/dj/lib/python3.6/site-packages/django/db/models/base.py", line 105, in __new__
</span><span> app_config = apps.get_containing_app_config(module)
</span><span> File "/home/nsukami/envs/dj/lib/python3.6/site-packages/django/apps/registry.py", line 237, in get_containing_app_config
</span><span> self.check_apps_ready()
</span><span> File "/home/nsukami/envs/dj/lib/python3.6/site-packages/django/apps/registry.py", line 124, in check_apps_ready
</span><span> raise AppRegistryNotReady("Apps aren't loaded yet.")
</span><span>django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
</span><span>(dj) ~/.../how_to_django/lwdj (master *%)
</span></code></pre>
<br>
<h4 id="installed-applications"><strong>Installed applications</strong></h4>
<p>I have not yet fully understood this part. <code>APP_LABEL</code> is actually containing the name of the current directory. By putting <code>APP_LABEL</code> inside <code>INSTALLED_APPS</code>, Django was able to find all the Django models existing inside the current directory:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>BASE_DIR = os.path.dirname(os.path.abspath(__file__))
</span><span>sys.path[</span><span style="color:#b5cea8;">0</span><span>] = os.path.dirname(BASE_DIR)
</span><span>APP_LABEL = os.path.basename(BASE_DIR)
</span><span>{{< /highlight >}}
</span><span>{{< highlight python >}}
</span><span> INSTALLED_APPS=[
</span><span> APP_LABEL,
</span><span> </span><span style="color:#d69d85;">'django.contrib.admin'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.auth'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.contenttypes'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.sessions'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.messages'</span><span>,
</span><span> </span><span style="color:#d69d85;">'django.contrib.staticfiles'</span><span>,
</span><span> </span><span style="color:#d69d85;">'rest_framework'</span><span>,
</span><span> ],
</span></code></pre>
<br>
<h4 id="app-label"><strong>App label</strong></h4>
<p>All our models were created outside of a <code>models.py</code> file. They're not even in a <em>real</em> Django application. So we were obliged to tell them which application they belongs to. This is done using the <a href="https://docs.djangoproject.com/en/1.10/ref/models/options/#app-label">app_label</a> attribute of the internal <code>class Meta</code> of Django models.
{{< highlight python >}}
class Author(models.Model):
name = models.CharField(max_length=200)
class Meta:
app_label = APP_LABEL</p>
<p>class Book(models.Model):
author = models.ForeignKey(Author, related_name='books')
title = models.CharField(max_length=400)
class Meta:
app_label = APP_LABEL
{{< /highlight >}}
<br><br></p>
<p>Really hope you learnt something. Also, I would not be able to write this post, without following the <em>StackTrace</em> leaved by the people who were there before me. A huge thanks to them:</p>
<ul>
<li><a href="http://dotmobo.github.io/django-minimaliste.html">Créer une application django minimaliste</a></li>
<li><a href="http://blog.fahhem.com/2011/10/django-models-without-apps-or-everything-django-truly-in-a-single-file/">Django Models without Apps OR Everything Django Truly in a Single File</a></li>
<li><a href="https://github.com/zenwalker/django-micro">Django as microframework</a></li>
</ul>
Make2016-11-25T00:00:00+00:002016-11-25T00:00:00+00:00
Unknown
https://nskm.xyz/posts/mk/<p>if you have some <strong>complex and esoteric shell commands</strong> that you have to run on a regular basis, it can be convenient to just package them up <em>somewhere</em>, so you don't have to <strong>remember them</strong> or <strong>type them out every time</strong>. <br><br></p>
<p>Why, I don't use Makefiles more often? Even though I know I should. One of the reason may be because, as an Emacs user, automating simple tasks is <em>funny</em> using <a href="https://www.gnu.org/software/emacs/manual/html_node/elisp/">Emacs Lisp</a>. Following are simple elisp functions I use to compile a C file, nothing extraordinary:</p>
<pre data-lang="lisp" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-lisp "><code class="language-lisp" data-lang="lisp"><span>(</span><span style="color:#569cd6;">defun </span><span>my-c-save-compile ()
</span><span> </span><span style="color:#d69d85;">"Save, and compile c file."
</span><span> (interactive)
</span><span> (save-buffer)
</span><span> (compile
</span><span> (format </span><span style="color:#d69d85;">"cc -Wall -Wextra -pedantic -std=c11 -g %s -o ./bin/%s"
</span><span> (buffer-file-name)
</span><span> (file-name-base (buffer-file-name)))))
</span><span>
</span><span>(</span><span style="color:#569cd6;">defun </span><span>my-c-run ()
</span><span> </span><span style="color:#d69d85;">"Run c code."
</span><span> (interactive)
</span><span> (compile (format </span><span style="color:#d69d85;">"./bin/%s"
</span><span> (file-name-base (buffer-file-name)))))
</span><span>
</span></code></pre>
<br>
<p>Another reason I don't use Makefiles more often could be, the <a href="http://www.tldp.org/LDP/Bash-Beginners-Guide/html/Bash-Beginners-Guide.html">Bourne Again Shell</a>. I find it easier to quickly write shell aliases for repetitive tasks. The following is some aliases I <em>sometimes</em> use whenever I'm doing <a href="https://www.djangoproject.com/">Django</a> development:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#608b4e;">#!/bin/sh
</span><span>
</span><span style="color:#608b4e;"># my weird aliases
</span><span>alias djo_runserver=</span><span style="background-color:#282828;color:#d69d85;">"python manage.py runserver"</span><span>
</span><span>alias djo_shell=</span><span style="background-color:#282828;color:#d69d85;">"python manage.py shell"</span><span>
</span><span>alias djo_shellp=</span><span style="background-color:#282828;color:#d69d85;">"python manage.py shell_plus"</span><span>
</span><span>alias djo_make=</span><span style="background-color:#282828;color:#d69d85;">"python manage.py makemigrations"</span><span>
</span><span>alias djo_mig=</span><span style="background-color:#282828;color:#d69d85;">"python manage.py migrate"</span><span>
</span><span>alias djo_csu=</span><span style="background-color:#282828;color:#d69d85;">"python manage.py createsuperuser"</span><span>
</span><span>alias djo_smtpd=</span><span style="background-color:#282828;color:#d69d85;">"python -m smtpd -n -c DebuggingServer localhost:1025"</span><span>
</span></code></pre>
<br>
<p>Make do have a reputation for being complex. Indeed, it can be very <a href="http://okmij.org/ftp/Computation/#Makefile-functional">esoteric</a>. The full <a href="https://www.gnu.org/software/make/manual/make.html">manual</a> is a 183 pages. Fortunately, you can ignore most of this, Make can be used in a very simple and incredibly useful way. <br><br></p>
<p>The following, is a Makefile I sometimes use in replacement of the shell aliases above:</p>
<pre data-lang="make" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#608b4e;"># Makefile for (the project)
</span><span style="color:#608b4e;">#
</span><span style="color:#608b4e;"># Nsukami
</span><span style="color:#608b4e;"># November 2016
</span><span>
</span><span>SHELL := </span><span style="background-color:#282828;color:#d69d85;">/bin/sh</span><span>
</span><span>PROJECT := </span><span style="background-color:#282828;color:#d69d85;">tas</span><span>
</span><span>PYTHON_BIN := </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">VIRTUAL_ENV</span><span style="background-color:#282828;color:#569cd6;">)</span><span style="background-color:#282828;color:#d69d85;">/bin</span><span>
</span><span>
</span><span>DEV_SETTINGS = </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">PROJECT</span><span style="background-color:#282828;color:#569cd6;">)</span><span style="background-color:#282828;color:#d69d85;">.settings.dev</span><span>
</span><span>PROD_SETTINGS = </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">PROJECT</span><span style="background-color:#282828;color:#569cd6;">)</span><span style="background-color:#282828;color:#d69d85;">.settings.prod</span><span>
</span><span>TEST_SETTINGS = </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">PROJECT</span><span style="background-color:#282828;color:#569cd6;">)</span><span style="background-color:#282828;color:#d69d85;">.settings.test</span><span>
</span><span>BASE_DIR = </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">shell dirname </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">realpath </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">lastword </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">MAKEFILE_LIST</span><span style="background-color:#282828;color:#569cd6;">))))</span><span>
</span><span>PMP = </span><span style="background-color:#282828;color:#d69d85;">python manage.py</span><span>
</span><span>
</span><span style="color:#608b4e;"># if not on prod server, use dev settings
</span><span style="color:#569cd6;">ifneq </span><span>(</span><span style="color:#569cd6;">$(</span><span>shell uname -n</span><span style="color:#569cd6;">)</span><span>,phobos)
</span><span> SETTINGS = </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">DEV_SETTINGS</span><span style="background-color:#282828;color:#569cd6;">)</span><span>
</span><span style="color:#569cd6;">else
</span><span> SETTINGS = </span><span style="background-color:#282828;color:#569cd6;">$(</span><span style="background-color:#282828;color:#dcdcdc;">PROD_SETTINGS</span><span style="background-color:#282828;color:#569cd6;">)</span><span>
</span><span style="color:#569cd6;">endif
</span><span>
</span><span>
</span><span style="color:#608b4e;"># target: all - Default target. Does nothing.
</span><span>all:
</span><span> </span><span style="color:#569cd6;">@</span><span>echo </span><span style="color:#d69d85;">"Nothing to do by default. Try 'make help'"
</span><span>
</span><span style="color:#608b4e;"># target: help - Show list of targets + description
</span><span>help:
</span><span> </span><span style="color:#569cd6;">@</span><span>egrep </span><span style="color:#d69d85;">"^# target:" </span><span style="color:#569cd6;">[</span><span>Mm</span><span style="color:#569cd6;">]</span><span>akefile
</span><span>
</span><span style="color:#608b4e;"># target: init - Install that a dev needs to get up and running.
</span><span>init: </span><span style="background-color:#282828;color:#d69d85;">bootstrap.sh</span><span>
</span><span> cd </span><span style="color:#569cd6;">$(</span><span>BASE_DIR</span><span style="color:#569cd6;">) && </span><span>sudo ./bootstrap.sh
</span><span>
</span><span style="color:#608b4e;"># target: clean - remove all useless files
</span><span>clean: </span><span style="background-color:#282828;color:#d69d85;">clean-pyc clean-build</span><span>
</span><span>
</span><span style="color:#608b4e;"># target: clean-pyc - remove all auto generated files
</span><span>clean-pyc:
</span><span> find . -name </span><span style="color:#d69d85;">'*.pyc'</span><span> -exec rm --force {} +
</span><span> find . -name </span><span style="color:#d69d85;">'*.pyo'</span><span> -exec rm --force {} +
</span><span> find . -name </span><span style="color:#d69d85;">'*~'</span><span> -exec rm --force {} +
</span><span>
</span><span style="color:#608b4e;"># target: clean-build - remove all auto generated folders
</span><span>clean-build:
</span><span> </span><span style="color:#569cd6;">-</span><span>rm --force --recursive build/
</span><span> </span><span style="color:#569cd6;">-</span><span>rm --force --recursive dist/
</span><span> </span><span style="color:#569cd6;">-</span><span>rm --force --recursive htmlcov
</span><span> </span><span style="color:#569cd6;">-</span><span>rm --force --recursive .coverage
</span><span> </span><span style="color:#569cd6;">-</span><span>rm --force --recursive </span><span style="color:#569cd6;">*</span><span>.egg-info
</span><span>
</span><span style="color:#608b4e;"># target: translate - calls the "makemessages" django command
</span><span>translate:
</span><span> cd </span><span style="color:#569cd6;">$(</span><span>BASE_DIR</span><span style="color:#569cd6;">) && $(</span><span>PMP</span><span style="color:#569cd6;">) </span><span>makemessages --settings=</span><span style="color:#569cd6;">$(</span><span>SETTINGS</span><span style="color:#569cd6;">)</span><span> -a
</span><span>
</span><span style="color:#608b4e;"># target: test - calls the "test" django command
</span><span>test:
</span><span> </span><span style="color:#569cd6;">$(</span><span>PMP</span><span style="color:#569cd6;">) </span><span>test --settings=</span><span style="color:#569cd6;">$(</span><span>TEST_SETTINGS</span><span style="color:#569cd6;">)
</span><span>
</span><span style="color:#608b4e;"># target: run - calls the "runserver" django command
</span><span>run:
</span><span> </span><span style="color:#569cd6;">$(</span><span>PMP</span><span style="color:#569cd6;">) </span><span>runserver --settings=</span><span style="color:#569cd6;">$(</span><span>SETTINGS</span><span style="color:#569cd6;">)
</span><span>
</span><span style="color:#608b4e;"># target: update - install (and update) pip requirements
</span><span>update:
</span><span> pip install -U -r </span><span style="color:#569cd6;">$(</span><span>BASE_DIR</span><span style="color:#569cd6;">)</span><span>/requirements/dev.txt
</span><span>
</span><span style="color:#608b4e;"># target: collect - calls the "collectstatic" django command
</span><span>collect:
</span><span> </span><span style="color:#569cd6;">$(</span><span>PMP</span><span style="color:#569cd6;">) </span><span>collectstatic --settings=</span><span style="color:#569cd6;">$(</span><span>SETTINGS</span><span style="color:#569cd6;">)</span><span> --noinput
</span><span>
</span><span style="color:#608b4e;"># target: rebuild - rebuild tables from models
</span><span>rebuild:
</span><span> </span><span style="color:#569cd6;">$(</span><span>PMP</span><span style="color:#569cd6;">) </span><span>makemigrations --settings=</span><span style="color:#569cd6;">$(</span><span>SETTINGS</span><span style="color:#569cd6;">)
</span><span> </span><span style="color:#569cd6;">$(</span><span>PMP</span><span style="color:#569cd6;">) </span><span>migrate --settings=</span><span style="color:#569cd6;">$(</span><span>SETTINGS</span><span style="color:#569cd6;">)
</span><span>
</span><span style="color:#608b4e;"># target: deploy - let make take care of deployment
</span><span>deploy:
</span><span> </span><span style="color:#608b4e;"># Should I call fabric here?
</span><span>
</span><span>
</span><span>.ONESHELL:
</span><span>.PHONY: </span><span style="background-color:#282828;color:#d69d85;">all help translate test clean update collect rebuild run clean-pyc clean-build clean init</span><span>
</span></code></pre>
<br>
<p>Finally, the other reason, and probably the main reason I don't use Makefiles more often, it's because of <a href="http://www.fabfile.org/">Fabric</a>. Following is a fabfile more or less doing the same thing as the Makefile above:</p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#608b4e;">#!/usr/bin/env python
</span><span style="color:#608b4e;"># -*- coding: utf-8 -*- #
</span><span>
</span><span style="color:#608b4e;">""" fabric commands """
</span><span>
</span><span style="color:#9b9b9b;">from </span><span>fabric.api </span><span style="color:#9b9b9b;">import </span><span>env, sudo, cd, local, task, run
</span><span style="color:#9b9b9b;">import </span><span>fabric.contrib.project </span><span style="color:#9b9b9b;">as </span><span>project
</span><span style="color:#9b9b9b;">import </span><span>os
</span><span>
</span><span style="color:#608b4e;"># Local path configuration (can be absolute or relative to fabfile)
</span><span>env.hosts = [</span><span style="color:#d69d85;">'localhost'</span><span>]
</span><span>env.use_ssh_config = </span><span style="color:#569cd6;">True
</span><span>env.ssh_config_path = </span><span style="color:#d69d85;">'/home/nsukami/.ssh/config'
</span><span>env.owner = </span><span style="color:#d69d85;">'nsukami'
</span><span>
</span><span>PROJECT = </span><span style="color:#d69d85;">"tas"
</span><span>
</span><span>DEV_SETTINGS = PROJECT + </span><span style="color:#d69d85;">".settings.dev"
</span><span>PROD_SETTINGS = PROJECT + </span><span style="color:#d69d85;">".settings.prod"
</span><span>TEST_SETTINGS = PROJECT + </span><span style="color:#d69d85;">".settings.test"
</span><span>BASE_DIR = os.path.dirname(os.path.abspath(__file__))
</span><span>PMP = </span><span style="color:#d69d85;">"python manage.py "
</span><span>
</span><span>
</span><span>
</span><span style="color:#569cd6;">def </span><span>uname():
</span><span> </span><span style="color:#608b4e;">""" Prints information about the host. """
</span><span> local(</span><span style="color:#d69d85;">"uname -n"</span><span>)
</span><span>
</span><span style="color:#569cd6;">if </span><span>uname() </span><span style="color:#569cd6;">is not </span><span style="color:#d69d85;">"phobos"</span><span>:
</span><span> SETTINGS = DEV_SETTINGS
</span><span style="color:#569cd6;">else</span><span>:
</span><span> SETTINGS = PROD_SETTINGS
</span><span>
</span><span>@task
</span><span style="color:#569cd6;">def </span><span>init():
</span><span> </span><span style="color:#d69d85;">"init - Install that a dev needs to get up and running."
</span><span> </span><span style="color:#569cd6;">with </span><span>cd(BASE_DIR):
</span><span> sudo(</span><span style="color:#d69d85;">"./bootstrap.sh"</span><span>)
</span><span>
</span><span>@task
</span><span style="color:#569cd6;">def </span><span>clean():
</span><span> </span><span style="color:#d69d85;">"clean - remove all useless files"
</span><span> clean_pyc()
</span><span> clean_build()
</span><span>
</span><span style="color:#569cd6;">def </span><span>clean_pyc():
</span><span> </span><span style="color:#d69d85;">"# clean-pyc - remove all auto generated files"
</span><span> local(</span><span style="color:#d69d85;">"find . -name '*.pyc' -exec rm --force </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;"> +"</span><span>)
</span><span> local(</span><span style="color:#d69d85;">"find . -name '*.pyo' -exec rm --force </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;"> +"</span><span>)
</span><span> local(</span><span style="color:#d69d85;">"find . -name '*~' -exec rm --force </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;"> +"</span><span>)
</span><span>
</span><span style="color:#569cd6;">def </span><span>clean_build():
</span><span> </span><span style="color:#d69d85;">"clean-build - remove all auto generated folders"
</span><span> local(</span><span style="color:#d69d85;">"rm --force --recursive build"</span><span>)
</span><span> local(</span><span style="color:#d69d85;">"rm --force --recursive dist"</span><span>)
</span><span> local(</span><span style="color:#d69d85;">"rm --force --recursive htmlcov"</span><span>)
</span><span> local(</span><span style="color:#d69d85;">"rm --force --recursive .coverage"</span><span>)
</span><span> local(</span><span style="color:#d69d85;">"rm --force --recursive *.egg-info"</span><span>)
</span><span>
</span><span>@task
</span><span style="color:#569cd6;">def </span><span>translate():
</span><span> </span><span style="color:#d69d85;">"translate - calls the 'makemessages' django command"
</span><span> </span><span style="color:#569cd6;">with </span><span>cd(BASE_DIR):
</span><span> cmd = PMP + </span><span style="color:#d69d85;">"makemessages --settings=</span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;"> -a"</span><span>.format(SETTINGS)
</span><span> local(cmd)
</span><span>
</span><span>@task
</span><span style="color:#569cd6;">def </span><span>test():
</span><span> </span><span style="color:#d69d85;">"test - calls the 'test' django command"
</span><span> cmd = PMP + </span><span style="color:#d69d85;">"test --settings=</span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">"</span><span>.format(SETTINGS)
</span><span> local(cmd)
</span><span>
</span><span>@task
</span><span style="color:#569cd6;">def </span><span>runserver():
</span><span> </span><span style="color:#d69d85;">"run - calls the 'runserver' django command"
</span><span> cmd = PMP + </span><span style="color:#d69d85;">"runserver --settings=</span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">"</span><span>.format(SETTINGS)
</span><span> local(cmd)
</span><span>
</span><span>@task
</span><span style="color:#569cd6;">def </span><span>update():
</span><span> </span><span style="color:#d69d85;">"update - install (and update) pip requirements"
</span><span> cmd = </span><span style="color:#d69d85;">"pip install -U -r </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">/requirements/dev.txt"</span><span>.format(BASE_DIR)
</span><span> local(cmd)
</span></code></pre>
<br>
<p>We see that Fabric <em>can</em> be used as a build tool even though it isn't one. It is easy to read and write, especially for a Python developer. The drawback may be, it does require Python to run. <em>Seriously, who cares?</em>
<br><br></p>
<p>New year resolution: <strong>using Makefiles more often.</strong>
<br><br>
More on the topic:</p>
<ul>
<li><a href="https://www.gnu.org/software/make/manual/make.html">Make</a></li>
<li><a href="https://news.ycombinator.com/item?id=5276339">Make alternatives</a></li>
<li><a href="https://en.wikipedia.org/wiki/List_of_build_automation_software">List of build automation software</a></li>
</ul>
How to create processes2016-08-22T03:40:17+00:002016-08-22T03:40:17+00:00
Unknown
https://nskm.xyz/posts/process-creation/<p>We've seen <a href="/on-processes.md">processes</a>, We've said lots of things about processes, but we did not talk about process creation. Let's see how to create a process in C, in <a href="https://golang.org/">Golang</a>, and in <a href="https://www.rust-lang.org/en-US/">Rustlang</a> <em><strong>Disclaimer:</strong> I'm not doing a comparison to find out which one is better than the other. I just want to explore how the same task can be done using different approaches.<br><br></em></p>
<h3 id="fork-and-exec"><strong>Fork and Exec:</strong></h3>
<p>UNIX processes are created using the fork() <a href="https://en.wikipedia.org/wiki/System_call">system call</a>. Fork is the primary method of process creation on Unix-like operating systems. Fork is the system call that the parent process uses to "divide" itself, to "fork" into two identical processes. The child process will be the exact copy of the parent except for the return value.<br></p>
<p>The <a href="https://en.wikipedia.org/wiki/Exec_(system_call)">exec()</a> system call and its family of functions are almost always used with fork(). The exec() syscall and the like overwrite the current [stackframe]({{< ref on-processes >}})with the name of the application passed to it.<br><br></p>
<h4 id="c"><strong>C:</strong></h4>
<p>In C, there is nothing special to do, contrary to Golang and Rustlang. You just really call fork(), then exec(), simple and direct. The following, is an example written in C:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><sys/types.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><unistd.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><sys/wait.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><string.h>
</span><span>
</span><span style="color:#569cd6;">void </span><span>fork_and_execute();
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> fork_and_execute();
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>fork_and_execute() {
</span><span> </span><span style="color:#569cd6;">int</span><span> pid, status;
</span><span> pid = fork();
</span><span>
</span><span> </span><span style="color:#608b4e;">// if pid equals 0,
</span><span> </span><span style="color:#569cd6;">if</span><span>(pid == </span><span style="color:#b5cea8;">0</span><span>){
</span><span> </span><span style="color:#608b4e;">// we're dealing with the child process
</span><span> printf(</span><span style="color:#d69d85;">"I am the child process, my pid is </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, getpid());
</span><span>
</span><span> </span><span style="color:#608b4e;">// the program we want to execute
</span><span> </span><span style="color:#569cd6;">char *</span><span>cmd[</span><span style="color:#b5cea8;">5</span><span>] = {</span><span style="color:#d69d85;">"ls"</span><span>, </span><span style="color:#d69d85;">"-a"</span><span>, </span><span style="color:#d69d85;">"-l"</span><span>, </span><span style="color:#d69d85;">"-h"</span><span>, </span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\0</span><span style="color:#d69d85;">'</span><span>};
</span><span>
</span><span> </span><span style="color:#608b4e;">// execvp will search for ls on $PATH
</span><span> </span><span style="color:#569cd6;">if </span><span>(execvp(</span><span style="color:#569cd6;">*</span><span>cmd, cmd) < </span><span style="color:#b5cea8;">0</span><span>){
</span><span> </span><span style="color:#608b4e;">// catch any error that may occur during execution
</span><span> printf(</span><span style="color:#d69d85;">"*** ERROR: exec failed</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> perror(</span><span style="color:#569cd6;">*</span><span>cmd);
</span><span> exit(EXIT_FAILURE);
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#608b4e;">// returned pid is -1, fork operation failed
</span><span> </span><span style="color:#569cd6;">if</span><span>(pid < </span><span style="color:#b5cea8;">0</span><span>){
</span><span> printf(</span><span style="color:#d69d85;">"*** ERROR: forking child process failed</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> perror(</span><span style="color:#d69d85;">"fork"</span><span>);
</span><span> exit(EXIT_FAILURE);
</span><span> }
</span><span>
</span><span> </span><span style="color:#608b4e;">// The wait system-call blocks the parent process
</span><span> </span><span style="color:#608b4e;">// and waits for the child-process to end.
</span><span> </span><span style="color:#569cd6;">while </span><span>(wait(</span><span style="color:#569cd6;">&</span><span>status) != pid){}
</span><span>}
</span></code></pre>
<br>
<h4 id="go"><strong>Go:</strong></h4>
<p>In Go, it is recommended to use the <a href="https://golang.org/pkg/os/exec/">exec</a> package. It runs external commands, and wraps <a href="https://golang.org/pkg/os/#StartProcess">os.StartProcess</a> function for you. Following, is the previous example, this time written in Golang:</p>
<pre data-lang="go" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#569cd6;">package </span><span>main
</span><span>
</span><span style="color:#569cd6;">import </span><span style="color:#d69d85;">"syscall"
</span><span style="color:#569cd6;">import </span><span style="color:#d69d85;">"os"
</span><span style="color:#569cd6;">import </span><span style="color:#d69d85;">"os/exec"
</span><span>
</span><span style="color:#569cd6;">func </span><span>main() {
</span><span> </span><span style="color:#608b4e;">// check if ls command exists in the PATH
</span><span> binary, lookErr := exec.LookPath(</span><span style="color:#d69d85;">"ls"</span><span>)
</span><span> </span><span style="color:#569cd6;">if </span><span>lookErr </span><span style="color:#569cd6;">!= nil </span><span>{
</span><span> panic(lookErr)
</span><span> }
</span><span>
</span><span> </span><span style="color:#608b4e;">// the program we want to execute
</span><span> args := []string{</span><span style="color:#d69d85;">"ls"</span><span>, </span><span style="color:#d69d85;">"-a"</span><span>, </span><span style="color:#d69d85;">"-l"</span><span>, </span><span style="color:#d69d85;">"-h"</span><span>}
</span><span>
</span><span> </span><span style="color:#608b4e;">// fork new process and execute our program
</span><span> execErr := syscall.Exec(binary, args, os.Environ())
</span><span>
</span><span> </span><span style="color:#608b4e;">// catch error if any
</span><span> </span><span style="color:#569cd6;">if </span><span>execErr </span><span style="color:#569cd6;">!= nil </span><span>{
</span><span> panic(execErr)
</span><span> }
</span><span>}
</span></code></pre>
<br>
<h4 id="rust"><strong>Rust:</strong></h4>
<p>In Rust, we have <a href="https://doc.rust-lang.org/std/process/struct.Command.html">std::process::Command</a>. It is is a type that acts as a process builder and provides fine-grained control over how the new process should be spawned. If we browse the <a href="https://doc.rust-lang.org/src/std/process.rs.html#521">source code</a>, we can read beginning line 521:</p>
<blockquote>
<p>Constructs a new <code>Command</code> for launching the program at</p>
</blockquote>
<blockquote>
<p>path <code>program</code>, with the following default configuration:</p>
</blockquote>
<blockquote>
</blockquote>
<blockquote>
<ul>
<li>No arguments to the program</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>Inherit the current process's environment</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>Inherit the current process's working directory</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>Inherit stdin/stdout/stderr for <code>spawn</code> or <code>status</code>, but create pipes for <code>output</code></li>
</ul>
</blockquote>
<br>
<p>Basically, it behave a little bit like a call to fork(), followed by a call to exec(). Let's rewrite the previous example in Rust:</p>
<pre data-lang="rust" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#569cd6;">use </span><span>std::process::Command;
</span><span>
</span><span style="color:#569cd6;">fn </span><span>main() {
</span><span> </span><span style="color:#569cd6;">let</span><span> output = Command::new(</span><span style="color:#d69d85;">"ls"</span><span>)
</span><span> .arg(</span><span style="color:#d69d85;">"-a"</span><span>)
</span><span> .arg(</span><span style="color:#d69d85;">"-l"</span><span>)
</span><span> .arg(</span><span style="color:#d69d85;">"-h"</span><span>)
</span><span> .output()
</span><span> .expect(</span><span style="color:#d69d85;">"ls command failed to start"</span><span>);
</span><span>
</span><span> println!(</span><span style="color:#d69d85;">"stdout: </span><span style="color:#b4cea8;">{}</span><span style="color:#d69d85;">"</span><span>, String::from_utf8_lossy(</span><span style="color:#569cd6;">&</span><span>output.stdout));
</span><span>}
</span></code></pre>
<br>
<p>The <a href="https://doc.rust-lang.org/std/process/struct.Command.html#method.output">output</a> function executes the command as a child process, waiting for it to finish and collecting all of its output. By default, stdin, stdout and stderr are captured (and used to provide the resulting output).<br></p>
<p>The <a href="https://doc.rust-lang.org/std/option/enum.Option.html#method.expect">expect</a> function is used for quick and dirty error handling.</p>
<br>
<h3 id="strace-trace-system-calls-and-signals"><strong>Strace, trace system calls and signals:</strong></h3>
<p>Somewhere in the strace's manual page, we can read:</p>
<blockquote>
<p>Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system calls.</p>
</blockquote>
<p>If everything went normally during the execution of our scripts, we should see, in the strace's output, lots of system calls, the most important ones being a call to <strong>clone()</strong>, followed by a call to <strong>exec()</strong>:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>...
</span><span>4714 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID</span><span style="color:#569cd6;">|</span><span>CLONE_CHILD_SETTID</span><span style="color:#569cd6;">|</span><span>SIGCHLD, child_tidptr=0x7fac5d28da90) = 4715
</span><span>...
</span><span>7564 execve(</span><span style="color:#d69d85;">"/bin/ls"</span><span>, </span><span style="color:#569cd6;">[</span><span style="color:#d69d85;">"ls"</span><span>, </span><span style="color:#d69d85;">"-a"</span><span>, </span><span style="color:#d69d85;">"-l"</span><span>, </span><span style="color:#d69d85;">"-h"</span><span style="color:#569cd6;">]</span><span>, </span><span style="color:#569cd6;">[</span><span>/</span><span style="color:#569cd6;">*</span><span> 74 vars </span><span style="color:#569cd6;">*</span><span>/</span><span style="color:#569cd6;">] </span><span><unfinished ...>
</span><span>...
</span></code></pre>
<br>
<p>fork() was the original UNIX system call. Only used to create new processes, not threads. Also, it is portable.</p>
<p>clone() is a new, versatile system call which can be used to create, depending on the passed option, a UNIX process, or a POSIX thread, or something in between, or something completely different. I invite you to <a href="http://man7.org/linux/man-pages/man2/clone.2.html">Read The Fabulous Manual</a> about clone().<br><br></p>
<p>I don't really know if there is something else you can read to learn more on this topic:</p>
<ul>
<li><a href="http://man7.org/linux/man-pages/man2/fork.2.html">Fork</a> manual page.</li>
<li><a href="http://man7.org/linux/man-pages/man3/exec.3.html">Exec</a> manual page.</li>
<li><a href="http://man7.org/linux/man-pages/man2/clone.2.html">Clone</a> manual page.</li>
<li><a href="http://man7.org/linux/man-pages/man1/strace.1.html">Strace</a> manual page.</li>
</ul>
Processes2016-08-09T03:40:17+00:002016-08-09T03:40:17+00:00
Unknown
https://nskm.xyz/posts/processes-introduction/<p>As a Debian user, I often want to see the active processes, sometimes, I want to kill them. I know how to launch a new process, inside my Python scripts. But what is a process?</p>
<br>
<p>A process is the execution context of a running program. When an executable program is read into memory by the kernel and executed, it becomes a process. Processes are dynamic entities, they are constantly changing as their machine code instructions are executed by the CPU.</p>
<br>
<p>With the exception of the <code>init process</code>, every process is the child of another process. And every process has the potential to be the parent of another process. The following is a small C program that will print the current process ID and the corresponding parent process ID:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><unistd.h>
</span><span>
</span><span style="color:#569cd6;">int </span><span>main (){
</span><span> </span><span style="color:#569cd6;">char</span><span> ch;
</span><span> printf (</span><span style="color:#d69d85;">"The process ID is </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">int</span><span>) getpid ());
</span><span> printf (</span><span style="color:#d69d85;">"The parent process ID is </span><span style="color:#b4cea8;">%d</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>, (</span><span style="color:#569cd6;">int</span><span>) getppid ());
</span><span> </span><span style="color:#569cd6;">while </span><span>( ch != </span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\r</span><span style="color:#d69d85;">' </span><span>&& ch != </span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">' </span><span>){
</span><span> ch = getchar();
</span><span> }
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span></code></pre>
<p>After execution, you should see something like:</p>
<pre data-lang="bash" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-bash "><code class="language-bash" data-lang="bash"><span>~
</span><span>+-> ./bin/ex1
</span><span>The process ID is 11147
</span><span>The parent process ID is 9552
</span></code></pre>
<p><a name="vas"></a></p>
<h3 id="virtual-address-space"><strong>Virtual address space?</strong></h3>
<p>A virtual address space, <code>VAS</code>, is the set of ranges of virtual addresses available to a process. The virtual address space is what a process sees when it executes. A process works with and only sees VAS. The operating system handles mapping VAS to real storage so that this is invisible to the running program.</p>
<br>
<p>We need VAS to prevent process A and process B from writing/reading in the same memory addresses. VAS is a way to achieve <a href="https://en.wikipedia.org/wiki/Process_isolation">process isolation</a>. With VAS, several programs could write to memory location at the same address without stepping over each others results.</p>
<br>
<center>
<img src="https://nskm.xyz/processed_images/vas-schema.9c317ee408a2f21e.png" />
</center>
<p><em>Image from <a href="http://www.tenouk.com/ModuleW_files/ccompilerlinker007.png">tenouk.com</a></em></p>
<br>
<h3 id="what-s-inside-vas"><strong>What's inside VAS?</strong></h3>
<p><code>VAS</code>, is the set of ranges of virtual addresses available to a process, so what is inside a VAS? A typical memory representation of process consists of following sections:</p>
<ol>
<li>
<p><strong>Stack:</strong><br>
VAS section that is used for several related purposes, the main one being to keep track of the point to which each active subroutine should return control when it finishes executing. The return address is pushed onto the stack by the caller. When it finishes, the called subroutine, pops the return address off the stack and transfers control to that address. <br><br>That section also stores temporary variables created by each function inside the program. The stack is a LIFO data structure, managed and optimized by the CPU quite closely. Every time a function declares a new variable, it is "pushed" onto the stack, you don't have to allocate the needed memory by hand. Then every time a function exits, all of the variables pushed onto the stack by that function, are freed for you. Once a stack variable is freed, that region of memory becomes available for other stack variables.</p>
</li>
<li>
<p><strong>Heap:</strong><br>
VAS section where dynamic memory allocation usually takes place. Unlike the Stack, this part of the VAS is not managed automatically for you, and is not as tightly managed by the CPU. You are in charge of allocating memory on the heap. You are also responsible for deallocating that memory once you don't need it any more. If you fail to do this, your program will have what is known as a memory leak.</p>
</li>
<li>
<p><strong>Initialized Data Segment:</strong><br>
Or simply, Data Segment. VAS section, which contains the global and static variables that are initialized by the programmer. The size of this segment is determined by the size of the values in the program's source code, and does not change at run time.</p>
</li>
<li>
<p><strong>Uninitialized (bss) Data Segment:</strong> <br>
VAS section that contains all global and static variables that are initialized to zero or do not have explicit initialization in the programer's source code.</p>
</li>
<li>
<p><strong>Text (code) Segment:</strong><br>
VAS section that contains machine instructions of the program. Those machine instructions can be thought of as the text of a novel: It tells the story of what the program does.</p>
</li>
</ol>
<br>
<h3 id="how-to-do-some-introspection"><strong>How to do some introspection?</strong></h3>
<p>How do we learn about the memory layout of a running process? How do we look at how program are laid out in memory?</p>
<ol>
<li><strong>Directly from /proc/PID/maps:<br></strong>
You can directly ask to the process information pseudo-filesystem: <a href="http://man7.org/linux/man-pages/man5/proc.5.html">proc</a>. Let's see the output of <code>cat /proc/6026/maps</code>. Here <code>6026</code> is the PID of a running process on my Debian:<br></li>
</ol>
<pre data-lang="txt" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-txt "><code class="language-txt" data-lang="txt"><span>00400000-00401000 r-xp 00000000 08:05 3181468 /home/nsukami/bin/how_to_vms
</span><span>00600000-00601000 rw-p 00000000 08:05 3181468 /home/nsukami/bin/how_to_vms
</span><span>01698000-016b9000 rw-p 00000000 00:00 0 [heap]
</span><span>7f747118a000-7f747132c000 r-xp 00000000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span>7f747132c000-7f747152b000 ---p 001a2000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span>7f747152b000-7f747152f000 r--p 001a1000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span>7f747152f000-7f7471531000 rw-p 001a5000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span>7f7471531000-7f7471535000 rw-p 00000000 00:00 0
</span><span>7f7471535000-7f7471555000 r-xp 00000000 08:01 130924 /lib/x86_64-linux-gnu/ld-2.19.so
</span><span>7f7471734000-7f7471737000 rw-p 00000000 00:00 0
</span><span>7f7471752000-7f7471755000 rw-p 00000000 00:00 0
</span><span>7f7471755000-7f7471756000 r--p 00020000 08:01 130924 /lib/x86_64-linux-gnu/ld-2.19.so
</span><span>7f7471756000-7f7471757000 rw-p 00021000 08:01 130924 /lib/x86_64-linux-gnu/ld-2.19.so
</span><span>7f7471757000-7f7471758000 rw-p 00000000 00:00 0
</span><span>7ffc4d89b000-7ffc4d8bc000 rw-p 00000000 00:00 0 [stack]
</span><span>7ffc4d99b000-7ffc4d99d000 r-xp 00000000 00:00 0 [vdso]
</span><span>7ffc4d99d000-7ffc4d99f000 r--p 00000000 00:00 0 [vvar]
</span><span>ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
</span></code></pre>
<br>
The Heap segment and the Stack segment are labelled, easy. The addresses are displayed from the lowest `00400000`, to the highest `ffffffffff601000`. The first segment, `00400000-00401000`, is read-only and executable, so, this is the Code segment. We know that between the Code segment, and the Heap segment, there is the Data segment. If you don't want to browse the `/proc` directory, there is also the [pmap](http://linux.die.net/man/1/pmap) command you can use to report the memory map of a process.
<br>
<ol start="2">
<li><strong>Using the GNU debugger:</strong>
GDB, the <em>awesome</em> <a href="https://www.gnu.org/software/gdb/">GNU Project debugger</a>, allows you, among others things, to see what is going on <code>inside</code> a running process. For example, from the gdb prompt, you can type <code>info proc mappings</code> to see the current address mapping:</li>
</ol>
<pre data-lang="txt" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-txt "><code class="language-txt" data-lang="txt"><span>process 8460
</span><span>Mapped address spaces:
</span><span>
</span><span> Start Addr End Addr Size Offset objfile
</span><span> 0x400000 0x401000 0x1000 0x0 /home/nsukami/bin/how_to_vms
</span><span> 0x600000 0x601000 0x1000 0x0 /home/nsukami/bin/how_to_vms
</span><span> 0x601000 0x622000 0x21000 0x0 [heap]
</span><span> 0x7ffff7a31000 0x7ffff7bd3000 0x1a2000 0x0 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span> 0x7ffff7bd3000 0x7ffff7dd2000 0x1ff000 0x1a2000 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span> 0x7ffff7dd2000 0x7ffff7dd6000 0x4000 0x1a1000 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span> 0x7ffff7dd6000 0x7ffff7dd8000 0x2000 0x1a5000 /lib/x86_64-linux-gnu/libc-2.19.so
</span><span> 0x7ffff7dd8000 0x7ffff7ddc000 0x4000 0x0
</span><span> 0x7ffff7ddc000 0x7ffff7dfc000 0x20000 0x0 /lib/x86_64-linux-gnu/ld-2.19.so
</span><span> 0x7ffff7fd7000 0x7ffff7fda000 0x3000 0x0
</span><span> 0x7ffff7ff5000 0x7ffff7ff8000 0x3000 0x0
</span><span> 0x7ffff7ff8000 0x7ffff7ffa000 0x2000 0x0 [vdso]
</span><span> 0x7ffff7ffa000 0x7ffff7ffc000 0x2000 0x0 [vvar]
</span><span> 0x7ffff7ffc000 0x7ffff7ffd000 0x1000 0x20000 /lib/x86_64-linux-gnu/ld-2.19.so
</span><span> 0x7ffff7ffd000 0x7ffff7ffe000 0x1000 0x21000 /lib/x86_64-linux-gnu/ld-2.19.so
</span><span> 0x7ffff7ffe000 0x7ffff7fff000 0x1000 0x0
</span><span> 0x7ffffffde000 0x7ffffffff000 0x21000 0x0 [stack]
</span><span>0xffffffffff600000 0xffffffffff601000 0x1000 0x0 [vsyscall]
</span></code></pre>
<br>
<p>And there is more to explore:</p>
<pre data-lang="txt" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-txt "><code class="language-txt" data-lang="txt"><span>(gdb) help info proc
</span><span>Show /proc process information about any running process.
</span><span>Specify any process id, or use the program being debugged by default.
</span><span>
</span><span>List of info proc subcommands:
</span><span>
</span><span>info proc all -- List all available /proc info
</span><span>info proc cmdline -- List command line arguments of the process
</span><span>info proc cwd -- List current working directory of the process
</span><span>info proc exe -- List absolute filename for executable of the process
</span><span>info proc mappings -- List of mapped memory regions
</span><span>info proc stat -- List process info from /proc/PID/stat
</span><span>info proc status -- List process info from /proc/PID/status
</span><span>
</span><span>Type "help info proc" followed by info proc subcommand name for full documentation.
</span><span>Type "apropos word" to search for commands related to "word".
</span><span>Command name abbreviations are allowed if unambiguous.
</span><span>(gdb)
</span></code></pre>
<br>
<h3 id="how-to-create-a-process"><strong>How to create a process?</strong></h3>
<p>1.<strong>fork():</strong>
The fork() system call is used to create processes. It takes no arguments and returns a process ID. The purpose of fork() is to create a new process, which becomes the child process of the caller:</p>
<pre data-lang="c" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdio.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><sys/types.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><unistd.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><stdlib.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><sys/wait.h>
</span><span style="color:#9b9b9b;">#include </span><span style="color:#d69d85;"><string.h>
</span><span>
</span><span style="color:#569cd6;">void </span><span>execute();
</span><span>
</span><span style="color:#569cd6;">int </span><span>main() {
</span><span> execute();
</span><span> </span><span style="color:#569cd6;">return </span><span style="color:#b5cea8;">0</span><span>;
</span><span>}
</span><span>
</span><span style="color:#569cd6;">void </span><span>execute() {
</span><span> </span><span style="color:#569cd6;">int</span><span> pid, status;
</span><span>
</span><span> </span><span style="color:#569cd6;">if</span><span>((pid = fork()) == </span><span style="color:#b5cea8;">0</span><span>){
</span><span> </span><span style="color:#569cd6;">char *</span><span>cmd[</span><span style="color:#b5cea8;">5</span><span>] = {</span><span style="color:#d69d85;">"ls"</span><span>, </span><span style="color:#d69d85;">"-a"</span><span>, </span><span style="color:#d69d85;">"-l"</span><span>, </span><span style="color:#d69d85;">"-h"</span><span>, </span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\0</span><span style="color:#d69d85;">'</span><span>};
</span><span>
</span><span> </span><span style="color:#569cd6;">if </span><span>(execvp(</span><span style="color:#569cd6;">*</span><span>cmd, cmd) < </span><span style="color:#b5cea8;">0</span><span>){
</span><span> printf(</span><span style="color:#d69d85;">"*** ERROR: exec failed</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> perror(</span><span style="color:#569cd6;">*</span><span>cmd);
</span><span> exit(</span><span style="color:#b5cea8;">1</span><span>);
</span><span> }
</span><span> }</span><span style="color:#569cd6;">else</span><span>{
</span><span> printf(</span><span style="color:#d69d85;">"*** ERROR: forking child process failed</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">"</span><span>);
</span><span> perror(</span><span style="color:#d69d85;">"fork"</span><span>);
</span><span> exit(EXIT_FAILURE);
</span><span> }
</span><span>
</span><span> </span><span style="color:#608b4e;">//The parent executes the wait.
</span><span> </span><span style="color:#569cd6;">while </span><span>(wait(</span><span style="color:#569cd6;">&</span><span>status) != pid){}
</span><span>}
</span></code></pre>
<p>I really would like to write more on this topic. Maybe in another post, otherwise the article will be too long, <em>and I have the dishes to do</em>. At least, we've seen that there is an interesting amount of informations behind what is called a process.</p>
<h3 id="i-hope-you-ve-learned-something-and-if-you-want-to-know-more">I hope you've learned something, and if you want to know more:</h3>
<ul>
<li><a href="http://www.tldp.org/LDP/tlk/kernel/processes.html">Processes</a></li>
<li><a href="https://www.kernel.org/doc/Documentation/filesystems/proc.txt">Proc</a></li>
<li><a href="http://man7.org/linux/man-pages/man5/proc.5.html">Proc man pages</a></li>
<li><a href="https://lwn.net/Articles/446528/">The "vsyscall" and "vDSO" segments</a></li>
<li><a href="https://www.gnu.org/software/gdb/">GDB</a></li>
<li><a href="http://turnoff.us/geek/the-jealous-process/">The jealous process</a></li>
</ul>
Network namespaces2016-07-08T00:00:00+00:002016-07-08T00:00:00+00:00
Unknown
https://nskm.xyz/posts/onn/<p>I use LXC containers mainly as isolated environments when I'm writing code, but also when I just want to try the last version of a package without being obliged to break something on my current configuration. This is the first part of a serie about LXC containers. In this post, I'll share with you what I've learned about network namespaces. Everything here will be done under Debian Jessie, as root, I'm using Python 3.5 and I've pip installed pyroute2.
<br><br></p>
<p>To keep the schemas as simple as possible, I won't mention the interface <code>lo</code> actually available inside the <code>root namespace</code>. The <code>root namespace</code> is the namespace available to you after a fresh boot of your Operating System. This is a small schema of my configuration showing you where we'll start: <img src="https://nskm.xyz/posts/onn/%7Bfilename%7D/images/root_ns.png" alt="Root namespace" />
<br><br></p>
<p>What we will do:</p>
<ol>
<li><a href="https://nskm.xyz/posts/onn/#namespace-creation">Create a namespace</a>.</li>
<li><a href="https://nskm.xyz/posts/onn/#veth-pair-creation">Create a virtual ethernet device pair</a>.</li>
<li><a href="https://nskm.xyz/posts/onn/#assign-on-side-to-namespace">Assign one side of the veth pair to the namespace</a>.</li>
<li><a href="https://nskm.xyz/posts/onn/#ip-addresses">Assign IP addresses to each side of the veth pair</a>.</li>
<li><a href="https://nskm.xyz/posts/onn/#default-route-for-namespace">Set the default route for our namespace</a>.</li>
<li><a href="https://nskm.xyz/posts/onn/#attach-veth-to-bridge">Attach the other side of the veth pair to a bridge</a>.
<br><br></li>
</ol>
<h4 id="network-namespaces">Network namespaces</h4>
<p>A <strong>namespace</strong> is a way of scoping a particular set of identifiers. You can use the same identifier multiple times in different namespaces. Most importantly, you can also restrict an identifier set visible to particular processes.
<br></p>
<p>Fact: the set of network interfaces and routing tables are shared across your entire Operating System. <a href="https://lwn.net/Articles/580893/">Network namespaces</a> change that fundamental assumption. With network namespaces, you can have different and separate instances of network interfaces and routing tables that operate independent of each other. Let's see how it works, using the Python package <a href="https://pypi.python.org/pypi/pyroute2">Pyroute2</a>:
<a name="namespace-creation"></a></p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>>>> </span><span style="color:#9b9b9b;">from </span><span>pyroute2 </span><span style="color:#9b9b9b;">import </span><span>IPDB, IPRoute, NetNS, netns, NSPopen
</span><span>>>> </span><span style="color:#9b9b9b;">from </span><span>subprocess </span><span style="color:#9b9b9b;">import </span><span>Popen, PIPE
</span><span>>>> ipdb_namespace1 = IPDB(nl=NetNS(</span><span style="color:#d69d85;">'namespace1'</span><span>))
</span><span>>>> nsp = NSPopen(</span><span style="color:#d69d85;">'namespace1'</span><span>, [</span><span style="color:#d69d85;">'ip'</span><span>, </span><span style="color:#d69d85;">'addr'</span><span>], stdout=PIPE)
</span><span>>>> </span><span style="color:#569cd6;">for </span><span>l </span><span style="color:#569cd6;">in </span><span>nsp.communicate()[</span><span style="color:#b5cea8;">0</span><span>].split(</span><span style="color:#569cd6;">b</span><span style="color:#d69d85;">'</span><span style="color:#e3bbab;">\n</span><span style="color:#d69d85;">'</span><span>):
</span><span style="color:#569cd6;">... </span><span>print(l)
</span><span style="color:#569cd6;">...
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#569cd6;">b</span><span style="color:#d69d85;">'1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default '
</span><span style="color:#569cd6;">b</span><span style="color:#d69d85;">' link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00'
</span><span style="color:#569cd6;">b</span><span style="color:#d69d85;">''
</span></code></pre>
<br>
<p>We've just created a network namespace named <code>namespace1</code>. The <code>ip addr</code> command launched inside the namespace <code>namespace1</code> tells us that there is only one interface, <code>lo</code>.
<img src="https://nskm.xyz/posts/onn/%7Bfilename%7D/images/ns_created.png" alt="Namespace1 created" />
<br><br></p>
<p>Now, we need a way to link our newly created namespace, to the <code>root namespace</code>. To do that, virtual ethernet devices, <code>veth pair</code>, need to be created and configured. A <code>veth pair</code> works like a patch cable, connecting two sides. It consists of two virtual interfaces, one of them is assigned to the root network namespace, while the other lives within the network namespace, <code>namespace1</code>:
<a name="veth-pair-creation"></a></p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>>>> ipdb_root = IPDB() </span><span style="color:#608b4e;"># root namespace
</span><span>>>> ipdb_root.create(ifname=</span><span style="color:#d69d85;">'v01'</span><span>, kind=</span><span style="color:#d69d85;">'veth'</span><span>, peer=</span><span style="color:#d69d85;">'vp01'</span><span>).commit()
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>{</span><span style="color:#d69d85;">'linkmode'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'broadcast'</span><span>: </span><span style="color:#d69d85;">'ff:ff:ff:ff:ff:ff'</span><span>, </span><span style="color:#d69d85;">'num_rx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'qdisc'</span><span>: </span><span style="color:#d69d85;">'noop'</span><span>, </span><span style="color:#d69d85;">'txqlen'</span><span>: </span><span style="color:#b5cea8;">1000</span><span>, </span><span style="color:#d69d85;">'ipaddr'</span><span>: [], </span><span style="color:#d69d85;">'group'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'flags'</span><span>: </span><span style="color:#b5cea8;">4098</span><span>, </span><span style="color:#d69d85;">'neighbours'</span><span>: [], </span><span style="color:#d69d85;">'mtu'</span><span>: </span><span style="color:#b5cea8;">1500</span><span>, </span><span style="color:#d69d85;">'promiscuity'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'vlans'</span><span>: [], </span><span style="color:#d69d85;">'address'</span><span>: </span><span style="color:#d69d85;">'ca:a0:d7:39:57:22'</span><span>, </span><span style="color:#d69d85;">'ipdb_scope'</span><span>: </span><span style="color:#d69d85;">'system'</span><span>, </span><span style="color:#d69d85;">'kind'</span><span>: </span><span style="color:#d69d85;">'veth'</span><span>, </span><span style="color:#d69d85;">'ipdb_priority'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ifi_type'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'family'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'carrier'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ports'</span><span>: [], </span><span style="color:#d69d85;">'peer'</span><span>: </span><span style="color:#d69d85;">'vp01'</span><span>, </span><span style="color:#d69d85;">'num_tx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'index'</span><span>: </span><span style="color:#b5cea8;">10</span><span>, </span><span style="color:#d69d85;">'ifname'</span><span>: </span><span style="color:#d69d85;">'v01'</span><span>, </span><span style="color:#d69d85;">'carrier_changes'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'operstate'</span><span>: </span><span style="color:#d69d85;">'DOWN'</span><span>}
</span><span>>>>
</span></code></pre>
<p><img src="https://nskm.xyz/posts/onn/%7Bfilename%7D/images/veth_created.png" alt="Veth pair device created" />
<br><br></p>
<p>Veth pair device created. Both sides of the veth pair device are actually in the <code>root namespace</code>. To link the <code>root namespace</code> to our newly created namespace <code>namespace1</code>, one side of the veth pair device, the peer, should be assigned to <code>namespace1</code>:
<a name="assign-on-side-to-namespace"></a></p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>>>> </span><span style="color:#569cd6;">with </span><span>ipdb_root.interfaces.vp01 </span><span style="color:#569cd6;">as </span><span>v:
</span><span style="color:#569cd6;">... </span><span>v.net_ns_fd = </span><span style="color:#d69d85;">'namespace1'
</span><span style="color:#569cd6;">...
</span><span>>>>
</span></code></pre>
<p><img src="https://nskm.xyz/posts/onn/%7Bfilename%7D/images/veth_peer_moved.png" alt="Veth peer moved" />
<br><br></p>
<p>Now, let's give IP addresses to all of our interfaces, and set everybody up:
<a name="ip-addresses"></a></p>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>>>> </span><span style="color:#569cd6;">with </span><span>ipdb_root.interfaces.v01 </span><span style="color:#569cd6;">as </span><span>veth:
</span><span style="color:#569cd6;">... </span><span>veth.add_ip(</span><span style="color:#d69d85;">'192.168.1.36/24'</span><span>)
</span><span style="color:#569cd6;">... </span><span>veth.up()
</span><span style="color:#569cd6;">...
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>{</span><span style="color:#d69d85;">'kind'</span><span>: </span><span style="color:#d69d85;">'veth'</span><span>, </span><span style="color:#d69d85;">'promiscuity'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'group'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ipdb_scope'</span><span>: </span><span style="color:#d69d85;">'system'</span><span>, </span><span style="color:#d69d85;">'peer'</span><span>: </span><span style="color:#d69d85;">'vp01'</span><span>, </span><span style="color:#d69d85;">'linkmode'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ipdb_priority'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'carrier_changes'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'address'</span><span>: </span><span style="color:#d69d85;">'f2:2d:aa:9d:16:da'</span><span>, </span><span style="color:#d69d85;">'mtu'</span><span>: </span><span style="color:#b5cea8;">1500</span><span>, </span><span style="color:#d69d85;">'carrier'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ifname'</span><span>: </span><span style="color:#d69d85;">'v01'</span><span>, </span><span style="color:#d69d85;">'num_tx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'index'</span><span>: </span><span style="color:#b5cea8;">6</span><span>, </span><span style="color:#d69d85;">'ifi_type'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'ports'</span><span>: [], </span><span style="color:#d69d85;">'vlans'</span><span>: [], </span><span style="color:#d69d85;">'ipaddr'</span><span>: [], </span><span style="color:#d69d85;">'qdisc'</span><span>: </span><span style="color:#d69d85;">'noop'</span><span>, </span><span style="color:#d69d85;">'neighbours'</span><span>: [], </span><span style="color:#d69d85;">'txqlen'</span><span>: </span><span style="color:#b5cea8;">1000</span><span>, </span><span style="color:#d69d85;">'broadcast'</span><span>: </span><span style="color:#d69d85;">'ff:ff:ff:ff:ff:ff'</span><span>, </span><span style="color:#d69d85;">'operstate'</span><span>: </span><span style="color:#d69d85;">'DOWN'</span><span>, </span><span style="color:#d69d85;">'num_rx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'flags'</span><span>: </span><span style="color:#b5cea8;">4098</span><span>, </span><span style="color:#d69d85;">'family'</span><span>: </span><span style="color:#b5cea8;">0</span><span>}
</span><span>>>>
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>>>> </span><span style="color:#569cd6;">with </span><span>ipdb_namespace1.interfaces.vp01 </span><span style="color:#569cd6;">as </span><span>veth:
</span><span style="color:#569cd6;">... </span><span>veth.add_ip(</span><span style="color:#d69d85;">'192.168.1.37/24'</span><span>)
</span><span style="color:#569cd6;">... </span><span>veth.up()
</span><span style="color:#569cd6;">...
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>{</span><span style="color:#d69d85;">'kind'</span><span>: </span><span style="color:#d69d85;">'veth'</span><span>, </span><span style="color:#d69d85;">'promiscuity'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'group'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ipdb_scope'</span><span>: </span><span style="color:#d69d85;">'system'</span><span>, </span><span style="color:#d69d85;">'ifi_type'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'linkmode'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ipdb_priority'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'carrier_changes'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'num_rx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'address'</span><span>: </span><span style="color:#d69d85;">'26:29:f4:06:8a:ea'</span><span>, </span><span style="color:#d69d85;">'mtu'</span><span>: </span><span style="color:#b5cea8;">1500</span><span>, </span><span style="color:#d69d85;">'carrier'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ifname'</span><span>: </span><span style="color:#d69d85;">'vp01'</span><span>, </span><span style="color:#d69d85;">'num_tx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'index'</span><span>: </span><span style="color:#b5cea8;">5</span><span>, </span><span style="color:#d69d85;">'ports'</span><span>: [], </span><span style="color:#d69d85;">'vlans'</span><span>: [], </span><span style="color:#d69d85;">'ipaddr'</span><span>: [], </span><span style="color:#d69d85;">'qdisc'</span><span>: </span><span style="color:#d69d85;">'noop'</span><span>, </span><span style="color:#d69d85;">'neighbours'</span><span>: [], </span><span style="color:#d69d85;">'txqlen'</span><span>: </span><span style="color:#b5cea8;">1000</span><span>, </span><span style="color:#d69d85;">'broadcast'</span><span>: </span><span style="color:#d69d85;">'ff:ff:ff:ff:ff:ff'</span><span>, </span><span style="color:#d69d85;">'operstate'</span><span>: </span><span style="color:#d69d85;">'DOWN'</span><span>, </span><span style="color:#d69d85;">'flags'</span><span>: </span><span style="color:#b5cea8;">4098</span><span>, </span><span style="color:#d69d85;">'family'</span><span>: </span><span style="color:#b5cea8;">0</span><span>}
</span><span>>>>
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>>>> </span><span style="color:#569cd6;">with </span><span>ipdb_namespace1.interfaces.lo </span><span style="color:#569cd6;">as </span><span>i:
</span><span style="color:#569cd6;">... </span><span>i.up()
</span><span style="color:#569cd6;">...
</span></code></pre>
<pre data-lang="python" style="background-color:#1e1e1e;color:#dcdcdc;" class="language-python "><code class="language-python" data-lang="python"><span>{</span><span style="color:#d69d85;">'flags'</span><span>: </span><span style="color:#b5cea8;">8</span><span>, </span><span style="color:#d69d85;">'promiscuity'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'group'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ipdb_scope'</span><span>: </span><span style="color:#d69d85;">'system'</span><span>, </span><span style="color:#d69d85;">'ifi_type'</span><span>: </span><span style="color:#b5cea8;">772</span><span>, </span><span style="color:#d69d85;">'linkmode'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'ipdb_priority'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'carrier_changes'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'address'</span><span>: </span><span style="color:#d69d85;">'00:00:00:00:00:00'</span><span>, </span><span style="color:#d69d85;">'mtu'</span><span>: </span><span style="color:#b5cea8;">65536</span><span>, </span><span style="color:#d69d85;">'carrier'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'ifname'</span><span>: </span><span style="color:#d69d85;">'lo'</span><span>, </span><span style="color:#d69d85;">'num_tx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'index'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'ports'</span><span>: [], </span><span style="color:#d69d85;">'vlans'</span><span>: [], </span><span style="color:#d69d85;">'ipaddr'</span><span>: [], </span><span style="color:#d69d85;">'qdisc'</span><span>: </span><span style="color:#d69d85;">'noop'</span><span>, </span><span style="color:#d69d85;">'neighbours'</span><span>: [], </span><span style="color:#d69d85;">'txqlen'</span><span>: </span><span style="color:#b5cea8;">0</span><span>, </span><span style="color:#d69d85;">'broadcast'</span><span>: </span><span style="color:#d69d85;">'00:00:00:00:00:00'</span><span>, </span><span style="color:#d69d85;">'operstate'</span><span>: </span><span style="color:#d69d85;">'DOWN'</span><span>, </span><span style="color:#d69d85;">'num_rx_queues'</span><span>: </span><span style="color:#b5cea8;">1</span><span>, </span><span style="color:#d69d85;">'family'</span><span>: </span><span style="color:#b5cea8;">0</span><span>}
</span><span>>>>
</span></code></pre>
<p><br><br></p>
<p>At this point, we can open a shell and directly check the configuration for <code>v01</code> interface:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>kintanu:~# ip addr show v01
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>6: v01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
</span><span> link/ether f2:2d:aa:9d:16:da brd ff:ff:ff:ff:ff:ff
</span><span> inet 192.168.1.36/24 scope global v01
</span><span> valid_lft forever preferred_lft forever
</span><span> inet6 fe80::f02d:aaff:fe9d:16da/64 scope link
</span><span> valid_lft forever preferred_lft forever
</span></code></pre>
<br>
<p>We can also, still from the shell, check the available interfaces inside <code>namespace1</code>:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>kintanu:~# ip netns exec namespace1 ip addr show
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
</span><span> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
</span><span> inet 127.0.0.1/8 scope host lo
</span><span> valid_lft forever preferred_lft forever
</span><span> inet6 ::1/128 scope host
</span><span> valid_lft forever preferred_lft forever
</span><span>5: vp01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
</span><span> link/ether 26:29:f4:06:8a:ea brd ff:ff:ff:ff:ff:ff
</span><span> inet 192.168.1.37/24 scope global vpeer01
</span><span> valid_lft forever preferred_lft forever
</span><span> inet6 fe80::2429:f4ff:fe06:8aea/64 scope link
</span><span> valid_lft forever preferred_lft forever
</span><span>kintanu:~#
</span></code></pre>
<br>
<p>A system uses its routing table to determine which network interface to use when sending packets to remote systems. Let's make all traffic leaving <code>namespace1</code> to go through <code>v01</code>:
<a name="default-route-for-namespace"></a></p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>kintanu:~# ip netns exec namespace1 ip route add default via 192.168.1.36
</span></code></pre>
<p><img src="https://nskm.xyz/posts/onn/%7Bfilename%7D/images/default-route-added.png" alt="Default route added" />
<br><br></p>
<p>I told you I often use LXC containers when I'm working. For that reason, I always have a bridge configured and up. Then, all the containers are connected to the internet, via the bridge, as <code>ports</code>. Let me show you:</p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>kintanu:~# ip addr show br0
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
</span><span> link/ether 3c:97:0e:65:c0:27 brd ff:ff:ff:ff:ff:ff
</span><span> inet 192.168.1.34/24 brd 192.168.1.255 scope global br0
</span><span> valid_lft forever preferred_lft forever
</span><span> inet6 fe80::3e97:eff:fe65:c027/64 scope link
</span><span> valid_lft forever preferred_lft forever
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>kintanu:~# brctl show
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>bridge name bridge id STP enabled interfaces
</span><span>br0 8000.3c970e65c027 no eth0
</span><span> vethLMRO1N
</span><span>kintanu:~#
</span></code></pre>
<br>
<p>The <code>br0</code> interface is the interface I use for going outside. In fact, <code>br0</code> is a bridge. <code>eth0</code> and <code>vethLMRON</code> are interfaces, <code>ports</code>, attached to that <a href="http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge">bridge</a>. <code>v01</code> should be attached to the <code>br0</code> bridge, otherwise, <code>v01</code> won't be able to send packets to the outside world:
<a name="attach-veth-to-bridge"></a></p>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>kintanu:~# brctl addif br0 v01
</span><span>kintanu:~# brctl show
</span></code></pre>
<pre style="background-color:#1e1e1e;color:#dcdcdc;"><code><span>bridge name bridge id STP enabled interfaces
</span><span>br0 8000.3c970e65c027 no eth0
</span><span> vethLMRO1N
</span><span> v01
</span><span>kintanu:~#
</span></code></pre>
<br>
<br><br>
If everything went fine, you should be able to ping an external host from `namespace1`:
```
kintanu:~# ip netns exec namespace1 ping 8.8.8.8
```
```
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 192.168.1.34: icmp_seq=1 Redirect Host(New nexthop: 192.168.1.1)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=104 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=105 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=105 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=104 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 104.190/104.859/105.375/0.595 ms
kintanu:~#
```
<br>
<p>We've just seen how to create a network namespace, and link that network namespace to the outside world. When playing with LXC containers, you will also heard about <code>control groups</code>, a kernel mechanism that will allow you to allocate resources among user-defined groups of tasks (processes) running on a system. That will be the topic of the next chapter.</p>
<p><br><br><br></p>
<p style="text-align: right;">All the diagrams were made using [Asciiflow](http://asciiflow.com/)</p>
Introduction2015-07-24T00:00:00+00:002015-07-25T00:00:00+00:00
Unknown
https://nskm.xyz/posts/intro/<p>Hello, I'm Patrick, born and raised in central <a href="#">Africa</a>, currently based in West Africa, <a href="#">Dakar Senegal</a>.</p>
<p>I am an experienced software engineer, passionate about <a href="#">free software</a>. I have a strong background in developing backend solutions. With almost 10 years of hands-on experience in the field, I have successfully contributed to <a href="#">free software</a> and I have successfully developed and deployed <a href="#">numerous</a> <a href="#">software</a> <a href="#">projects</a> <a href="#">across</a> <a href="#">various</a> <a href="#">industries</a>.</p>
<p>My expertise lies in <a href="#">free software</a>, <a href="#">Python programming</a>, <a href="#">databases</a>; enabling me to tackle complex challenges and deliver high-quality, scalable software solutions. I thrive in dynamic and collaborative environments, working closely with cross-functional teams to understand requirements, design architectures, and implement robust solutions.</p>
<p>Throughout my career, I have demonstrated a proven track record of delivering projects on time and within budget, while ensuring adherence to best practices and industry standards. I am adept at analyzing existing systems, identifying areas for improvement, and implementing enhancements that optimize performance and user experience.</p>
<p>Over the course of my professional journey, I have had the privilege of <a href="#">working as a programming instructor</a> for diverse groups of learners, ranging from beginners to experienced professionals. My <a href="#">teaching</a> approach is centered around creating an engaging and supportive <a href="#">learning environment</a>, where students can grasp complex concepts with ease and apply them effectively.</p>
<p>Beyond technical skills, I am a team player, strong communicator and effective problem solver. I excel at sharing knowledge, at translating complex technical concepts into clear & non-technical language, fostering effective communication between stakeholders, clients, and development teams.</p>
<p>I am constantly exploring emerging technologies and <a href="#">staying up-to-date</a> with industry trends, as I believe in continuous learning and professional growth. I am eager to leverage my skills and expertise to contribute to a dynamic organization, driving innovation and delivering exceptional software solutions.</p>