Vectors

2026-02-26T00:00:00+00:00

Disclaimers :</h2>

Opinions: The views expressed here are my own unless otherwise specified. They do not represent any employer, organization, or affiliated group. </li>

Code examples: All code examples are educational. Production use requires careful consideration of performance, security, and your specific data characteristics. </li>

Model choice: The `all-MiniLM-L6-v2</code> model is a practical default, but embedding model selection should depend on your domain. Consider fine-tuning for specialized use cases. </li>`
`This article is for: Django developers curious about semantic search and vector embeddings, or anyone wanting to understand how vectors fit alongside traditional structured data. </li>`
`This article was written on a Framework laptop 16, yeah! </li> </ol>`**Hook</h2> When I start a new Django</a> project, the instinct is immediate and familiar: open models.py</code> and start writing classes</a>. I think about my entity, its attributes, its relationships, and I translate that mental model almost directly into Python. That's great. It works. Since so many years. It's readable. It's what Django was built for. Amazing. Perfect. Flawless. If you follow my amazing and colossal work, you know that since more that two years now, I'm working on a project in the public health sector. I'm handling lots and lots and lots of data. I have to answer interesting questions like... Yet, I'm still reprensenting my data using classes and attributes. Reflexes. There's another way to represent data that's been quietly powering search engines, recommendation systems, and AI applications for years, it's increasingly accessible to application developers like me. I should learn to use it more often. Instead of describing data as a collection of named attributes, I can describe it as a point in a high-dimensional space, a vector</a>. This article walks through what that actually means, when it matters, and how to implement both approaches side by side in a real Django project. To increase your chances of understanding, it is recommended that you listen to this track</a> while reading the article. Enjoy! ToC</h2> The classical approach</a></li> The vector approach</a></li> Why this is not just for AI</a></li> Both approaches in Django</a></li> Going further</a></li> Representation models</a></li> How to practice</a></li> </ol> The classical approach: data as structure</h2> When you model a medical facility in a surveillance platform</a>, you probably write something like this: class HealthFacility(models.Model): name = models.CharField(max_length=255) region = models.CharField(max_length=100) facility_type = models.CharField(max_length=50) # hospital, clinic, health_post capacity = models.IntegerField() services = models.ManyToManyField('Service') </code></pre> This is structured representation. Data lives in named slots. Querying it means matching those slots exactly: # Find clinics in Dakar with pediatric services HealthFacility.objects.filter( region="Dakar", facility_type="clinic", services__name="pediatrics" ) </code></pre> This is powerful for what it is. You know exactly what you're looking for, the schema enforces consistency, and SQL</a> is extraordinarily good at these kinds of exact, predicate-based lookups. The limitation reveals itself the moment your questions become fuzzy. What if you want facilities that are similar to a given one — without being able to define exactly what "similar" means? What if a user types "district hospital with maternal care near the coast" and you need to find the best match? Predicate-based filtering breaks down because similarity isn't a predicate. It's a geometry problem. The vector approach: data as position</h2> A vector is simply an ordered list of numbers: [0.82, 0.14, 0.67, 0.03, 0.91, ...] </code></pre> The idea is that you can encode the meaning or characteristics of any piece of data as a point in a high-dimensional space, such that things which are semantically similar end up close together, and things that are different end up far apart. This is not hand-crafted feature engineering (though it can be). Modern embedding models</a> — neural networks trained on massive datasets — can take a sentence, a document, even an image, and produce a dense vector that captures its semantic content. The model has learned, from context, what concepts relate to each other. "Maternal health clinic" and "obstetric care center" will produce vectors that are very close. "Warehouse" will be far away. The distance between two vectors is typically measured with cosine similarity</a>: similarity = cos(θ) = (A · B) / (||A|| × ||B||) </code></pre> A value of 1 means identical direction (semantically equivalent). A value of 0 means orthogonal (unrelated). This single number replaces the entire predicate matching logic. Why this is not just for AI projects</h2> You might think: "I don't need semantic search. I need to store patient records." That's fair. This is exactly how I've done it on the 4S project</a>. But I'm more and more facing these new types of scenarios: Duplicate detection. Two users register "Hôpital Principal de Dakar" and "hopital principal dakar". Exact matching fails. Vector similarity catches them as near-duplicates. Recommendation. "Users who viewed this report also found these relevant." You don't need a collaborative filtering setup — you can just find the reports whose vectors are closest to the one currently being read. Free-text search that actually works. Users don't search with your field names. They type natural language. Vectors let you match that intent.... This one stung me very badly recently. Clustering without labels. You have 10,000 epidemiological case reports. You want to discover natural groupings without writing filtering rules. Find neighborhoods in vector space.... We are working in this particular topic. I look at how we are implementing this feature and I am already disgusted by the direction we are taking. Marvelous. Anomaly detection. A new weekly report has a vector that is very far from all historical ones. Flag it for review. Both approaches in Django: a concrete example</h2> Let's build a very small project that models epidemiological reports using both approaches. We'll use pgvector</a>, a PostgreSQL extension</a>, so vectors live right next to our traditional Django models. No separate vector databases</a>, no need for new infrastructure if you're already on the amazing Postgres</a>. Setup</h3> pip install django pgvector sentence-transformers psycopg2-binary # easy peasy </code></pre> Enable the extension in a Django migration: from pgvector.django import VectorExtension class Migration(migrations.Migration): operations = [ VectorExtension(), ] </code></pre> The traditional model</h3> # surveillance/models.py from django.db import models from pgvector.django import VectorField class EpiReport(models.Model): """ Classical structured representation. Every piece of information lives in a named, typed field. """ title = models.CharField(max_length=255) region = models.CharField(max_length=100) disease = models.CharField(max_length=100) week = models.IntegerField() year = models.IntegerField() case_count = models.IntegerField(default=0) severity = models.CharField( max_length=20, choices=[("low", "Low"), ("moderate", "Moderate"), ("high", "High")] ) summary = models.TextField() # The vector representation lives alongside the structured one. # 384 dimensions is the output size of the model we'll use. embedding = VectorField(dimensions=384, null=True, blank=True) class Meta: indexes = [ # Traditional index for structured queries models.Index(fields=["region", "disease", "year"]), ] def str(self): return f"{self.title} ({self.region}, W{self.week}/{self.year})" </code></pre> Note that both representations coexist in the same table. This is intentional — we're not replacing one with the other. We're giving ourselves two different query interfaces into the same data. Generating the embedding</h3> # surveillance/embeddings.py from sentence_transformers import SentenceTransformer # All-MiniLM-L6-v2 is small (80MB), fast, and produces 384-dim vectors. # It runs on CPU without issue, which matters in resource-constrained environments. # See here: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 _model = None def get_model(): global _model if _model is None: _model = SentenceTransformer("all-MiniLM-L6-v2") return _model def embed_report(report: "EpiReport") -> list[float]: """ Convert a report's semantic content into a vector. The text we embed is a natural language description of the report, so the model can capture the meaning, not just the keywords. """ text = ( f"{report.title}. " f"Disease: {report.disease}. " f"Region: {report.region}. " f"Severity: {report.severity}. " f"{report.summary}" ) model = get_model() return model.encode(text).tolist() </code></pre> The two query styles, side by side</h3> # surveillance/queries.py from pgvector.django import CosineDistance from .models import EpiReport from .embeddings import get_model # ── Approach 1: Structured Query ──────────────────────────────────────────── def get_high_severity_reports(region: str, disease: str, year: int): """ Classical Django ORM query. Fast, precise, requires you to know exactly what you want. """ return EpiReport.objects.filter( region=region, disease=disease, year=year, severity="high", ).order_by("-week") # ── Approach 2: Semantic Vector Query ─────────────────────────────────────── def find_similar_reports(query: str, top_k: int = 5): """ Vector similarity search. The query is natural language. No schema knowledge required. Returns the most semantically similar reports. """ model = get_model() query_vector = model.encode(query).tolist() return ( EpiReport.objects .annotate(distance=CosineDistance("embedding", query_vector)) .order_by("distance")[:top_k] ) def find_similar_to_report(report: EpiReport, top_k: int = 5): """ Given an existing report, find others that are semantically similar. Useful for "related reports" features or duplicate detection. """ return ( EpiReport.objects .exclude(pk=report.pk) .annotate(distance=CosineDistance("embedding", report.embedding)) .order_by("distance")[:top_k] ) </code></pre> Wiring it into views</h3> # surveillance/views.py from django.http import JsonResponse from .models import EpiReport from .queries import get_high_severity_reports, find_similar_reports def structured_search(request): """Traditional form-based search.""" reports = get_high_severity_reports( region=request.GET.get("region", ""), disease=request.GET.get("disease", ""), year=int(request.GET.get("year", 2024)), ) return JsonResponse({ "approach": "structured", "results": list(reports.values("title", "region", "disease", "severity")) }) def semantic_search(request): """Free-text semantic search.""" query = request.GET.get("q", "") reports = find_similar_reports(query, top_k=5) return JsonResponse({ "approach": "vector", "query": query, "results": [ { "title": r.title, "region": r.region, "disease": r.disease, "similarity": round(1 - r.distance, 3), # convert distance to similarity } for r in reports ] }) </code></pre> Now try these two requests: GET /search/structured/?region=Dakar&disease=cholera&year=2024 GET /search/semantic/?q=waterborne disease outbreaks in coastal areas </code></pre> The structured search finds exactly what you specified. The semantic search finds what you meant, even if those exact words don't appear anywhere in your data. The world is beautiful. When to use which</h2> On top of my tiny experience, and as a software engineer who has never used this technique on on production, I would say the right answer for most systems is both. And pgvector makes that even more practical without adding infrastructure complexity. Lean on structured queries when you need exactness: filtering by date range, counting cases, producing regulatory reports, joining across tables. SQL is unbeatable here. Lean on vector queries when you need semantic understanding: search boxes, recommendation engines, duplicate detection, anomaly flagging, clustering, or any time a user's intent can't be reduced to a dropdown value. Even more interesting, hybrid search: use structured filters to narrow the candidate set, then rank the survivors by vector similarity: def hybrid_search(region: str, query: str, top_k: int = 5): """ Filter structurally, then rank semantically. This is typically the most useful approach in production. """ model = get_model() query_vector = model.encode(query).tolist() return ( EpiReport.objects .filter(region=region) # hard filter: only this region .annotate(distance=CosineDistance("embedding", query_vector)) .order_by("distance")[:top_k] ) </code></pre> Going further</h2> Once you've internalized the two-lens model, here's where to take it next: 1. Fine-tune your own embedding model on domain data</h3> The all-MiniLM-L6-v2</code> model</a> was trained on general English text. It has no idea that "paludisme" and "malaria" are the same thing, or that "cas suspects" and "suspected cases" are equivalent in my context. Training a domain-specific embedding model on my own epidemiological corpus — even with a small dataset using contrastive learning — would make similarity search dramatically more meaningful. 2. Retrieval-augmented generation (RAG) on top of our vector store</h3> Once we have a vector store of reports, we're one step away from a system that can answer questions about your data in natural language. A user asks: "What were the major respiratory disease trends in Guinea-Bissau in 2023?" — our system retrieves the most relevant reports by vector similarity, then feeds them as context to an LLM</a> that synthesizes a coherent answer. 3. Vectors for anomaly detection in surveillance data</h3> Every week a new batch of case reports comes in. We</a> can embed each report and compare it to the historical distribution of vectors for that region and disease. A report that lands far from the cluster is a statistical anomaly, potentially a real outbreak signal</a>. This turns our vector store into a passive early-warning system without writing any explicit outbreak-detection rules. 4. Multimodal embeddings</h3> Text isn't the only thing we can embed. We can embed laboratory results, genomic sequences, geographic coordinates combined with case counts, even scanned paper forms via vision models. The moment we have multiple modalities in the same vector space, we can do cross-modal search: find reports whose geographic spread pattern resembles this map image. placeholder image for brain explosion 5. The theory underneath: why vector spaces work</h3> Representation models: other ways to represent data</h2> Structured attributes and vectors are two lenses. And there are other ways to represent data: Representation</th> Best question it answers</th> Tools</th> </tr> Class + attributes</td> What does this thing have?</td> Django ORM, PostgreSQL</td> </tr> Vector</td> What is this thing like?</td> pgvector, sentence-transformers</td> </tr> Graph</td> How does this thing connect?</td> Neo4j, networkx</td> </tr> Event stream</td> How did this thing become what it is?</td> Kafka, EventStoreDB</td> </tr> Logic / facts</td> What can we conclude about this thing?</td> Prolog, Datalog, RDF/OWL</td> </tr> Probability</td> How certain are we about this thing?</td> PyMC, Stan</td> </tr> Spatial</td> Where is this thing, relative to what?</td> PostGIS, GeoDjango</td> </tr> Tensor</td> How does this vary across multiple dimensions?</td> NumPy, PyTorch, Xarray</td> </tr> </table> How to practice</h2> You don't need production data to start experimenting with vectors and embeddings: Local development: Chroma</a>: Lightweight vector database that runs locally or in Docker.</li> pgvector in Docker</a>: A local PostgreSQL with pgvector extension in seconds.</li> Sentence Transformers Models</a>: Experiment with domain-specific models beyond all-MiniLM-L6-v2.</li> </ul> Learning resources: Hugging Face Hub</a>: 100k+ pre-trained models. Read papers, compare benchmark results.</li> Embedding Models Course</a>: Free, practical introduction to embeddings.</li> pgvector Documentation</a>: All indexing strategies and similarity operators explained.</li> </ul> More on this topic</h2> As a Django engineer, my mental model of "data as class with attributes" is solid and won't stop serving me</a>. But it's worth internalizing that it's one of at least two valid representations. I should definitely start to think of my data as having a position in semantic space, a whole class of problems (search, similarity, anomaly, clustering) will become natural rather than bespoke. Here we are, we wanted a banana but we received a gorilla holding the banana and the entire jungle. Here are some links to help you (and me, let's be honest) go further and tame the beast: The hidden power behind AI & search</a></li> Fine-tuning sentence transformers</a></li> Vector database comparison</a></li> Contrastive learning explained</a></li> Retrieval-Augmented Generation (RAG)</a></li> pgvector performance tuning</a></li> On the foolishness of "natural language programming"</a></li> </ul> If you found this article useful, please like, share and subscribe. While I still have your attention</h2> The Norwegian Consumer Council is an independent, governmentally funded organisation that advocates for consumer’s rights. It should be easy for consumers to make sustainable choices every day. Consumers have the right to be protected against exploitation – both financially and digitally. To ensure this, we work to provide easy access to information, enforceable rights, and sufficient redress options when something goes wrong. Digital products and services keep getting worse. In the new report Breaking Free: Pathways to a fair technological future, the Norwegian Consumer Council has delved into enshittification</a> and how to resist it. The report shows how this phenomenon affects both consumers and society at large, but that it is possible to turn the tide.</a> Geez, I would have loved to see African leaders do the same.... AWS (Day 1) 2026-02-02T00:00:00+00:00 Disclaimers :</h2> Opinions expressed in this post (and in any of all my posts) are solely, unless otherwise specified, those of the authors, me. Those opinions absolutely do not reflect the views, policies, positions of any organizations, employers, affiliated groups. </li> I don't agree with this, but my employer says I don't have the right to share the source code I've written in the course of my work. I don't ever want a problem</a>. So, I'll try to say as much as I can without divulging any specific information. </li> Writing acts as a powerful therapeutic outlet, allowing emotions such as anger and irritation to be released. </li> If you are a colleague and recognize the situation described below, remember: I'm critiquing the system, not the people. </li> Just in case my honesty lands me in hot water, my resume is here</a>! </li> This article is written primarily for Tidiane and Dethie. </li> I've strived for accuracy throughout this piece, but if you catch any errors, please reach out—I'd be grateful for the feedback and happy to make updates! </li> Render unto Caesar what is Caesar's, Claude helped me find up-to-date documentation links and he also helped beautify all the diagrams used (and not used) in this article. </li> </ol> Hook</h2> One of the joys of corporate life: you're politely asked what training would help you be more productive, and a few months later, you're enrolled in something completely unrelated to your request. I've been asked to take a course called "AWS Automation and Observability." Of course, I only have one choice: accept. Even though I don't need it. Even though I find it thoroughly boring. Even though I consider the whole thing unnecessary. Being forced into training you didn't choose (especially when you already have solid technical skills) is frustrating. It's even more frustrating when the content feels dated; someone in 2026 teaching Jenkins</a> as if it's revolutionary feels like being taught to use a rotary phone. There's a certain irony in being told: « Il est strictement interdit d'enregistrer, de filmer ou de fixer par quelque moyen que ce soit les sessions de formation. Les supports doivent rester confidentiels. » </blockquote> ...when the material itself is widely available online and hardly groundbreaking, when (at least for day 0), there was no custom material given by the instructor. I respect the instructor's knowledge of the subject, he clearly knows his stuff, but the format, the delivery, and the relevance left much to be desired. Perhaps the problem isn't the instructor but the constraints he was working under: limited time, the fact that there were too many people with different levels in the same class or budget restrictions. Still, mandatory training should be engaging, current, and actually useful. This was none of those things. Since there's no escaping it, might as well make the most of it and stay in a good mood</a>. Let's talk about what I wish the first day had covered and how it should have been covered. AWS</h2> Intro</h3> Amazon Web Services (AWS)</a> is Amazon's cloud computing platform, launched in 2006 with two initial services: S3 (Simple Storage Service)</a> and EC2 (Elastic Compute Cloud)</a>. The idea emerged from Amazon's internal infrastructure challenges—they had built robust, scalable systems to handle their e-commerce peaks, and realized they could rent that infrastructure to others. Today, AWS offers 200+ services</a> spanning compute, storage, databases, machine learning, networking, and more. It holds roughly 31% of the global cloud market share (as of 2024), making it the largest cloud provider. Phew. In Africa, the Cape Town region (af-south-1)</a> was launched in 2020. It is AWS's first and currently only African region. There's no dedicated region yet for West Africa specifically. Users typically connect to eu-west-3 (Paris) or af-south-1 (Cape Town) depending on latency requirements. AWS does also have : Edge locations (CloudFront POPs) in Lagos and Nairobi</a> for content delivery.</li> Wavelength zone in Dakar Senegal</a>, a new type of infrastructure designed to run workloads that require low latency or edge resiliency.</li> </ul> Alternatives: the hyperscalers</h3> Hyperscalers are cloud providers operating at massive scale; we are talking about data centers across multiple continents, serving millionZ of customers simultaneously. The "Big 3" dominate the market: Provider </th> Share</th> Strengths</th> Best For</th></tr></thead> AWS</a></td> ~31%</td> Widest service catalog, mature ecosystem</td> General purpose, enterprise, startups</td></tr> Azure</a></td> ~24%</td> Windows integration, hybrid cloud</td> Microsoft shops, hybrid deployments</td></tr> GCP</a></td> ~11%</td> Data analytics, ML/AI (TensorFlow), Kubernetes </td> Data-heavy workloads, ML projects</td></tr> </tbody></table> Notable alternatives: DigitalOcean</a> / Linode</a> / Vultr</a> — Simpler, developer-friendly, cheaper for small workloads</li> OVHcloud</a> — European provider, good for GDPR compliance, competitive pricing</li> Scaleway</a> — French provider, strong EU presence, interesting ARM offerings</li> Hetzner</a> — German provider, excellent price-to-performance ratio, popular for self-hosted projects</li> </ul> The hyperscaler choice often comes down to existing tooling, team expertise, and specific service needs rather than raw capability. They all solve similar problems. As an advocate for the free software movement</a>, what is my advice ? GCP: Although proprietary & evil, I should admit, Google has historically opened up key technologies</a> such as Kubernetes and many others</a> to promote interoperability and encourage customers to leave more closed competitors. Red Hat OpenShift (on any cloud)</a>: Not a hyperscaler</a> per se, but using this platform on a cloud infrastructure allows you to maintain a free and portable software layer between different providers. Why would a biomedical research center use AWS ?</h3> For an organization handling sensitive health data and research workloads: Compliance: HIPAA, GDPR-ready configurations</li> Burst capacity: spin up hundreds of CPUs for genomic analysis, pay only for what you use</li> Global collaboration: share datasets across continents with proper access controls</li> Focus on research: less time managing servers, more time on science</li> </ul> Table of contents</h2> Fundamentals</a> - Where things are (Regions, Availability Zones, Organizations)</li> Identity and Access Management</a> - Who can do what (IAM)</li> Network</a> - How things connect VPC basics (CIDR, Subnets)</li> Gateways (IGW, NAT)</li> Hybrid connectivity (VPN, Direct Connect)</li> Network Security (Security Groups, NACLs)</li> </ul> </li> Load Balancing</a> - Traffic distribution</li> Compute Services</a> - Where code runs</li> Storage</a> - Where data lives</li> Observability</a> - How to monitor</li> </ol> AWS concepts → Traditional infrastructure</h2> If you've been managing Linux servers, you already know most of these concepts. AWS just wraps them in managed services with different names: AWS Service/Concept</th> Traditional Equivalent</th> What's Different?</th></tr></thead> EC2</td> Physical server, VM (KVM/VMware)</td> Rent by the hour, no hardware to buy</td></tr> VPC</td> Network subnet, VLAN</td> Software-defined, fully isolated per customer</td></tr> Security Groups</td> iptables</code> rules (instance-level)</td> Stateful, attached to network interfaces</td></tr> Network ACLs</td> iptables</code> at router/subnet level</td> Stateless, numbered rules, processed in order</td></tr> S3</td> File server (NFS, Samba), MinIO</td> Object storage (not filesystem), infinite scale, HTTP API</td></tr> EBS</td> Hard drive, LVM volume</td> Network-attached block storage, point-in-time snapshots</td></tr> EFS</td> NFS share</td> Managed NFS, auto-scaling, multi-AZ</td></tr> RDS</td> PostgreSQL server you installed</td> AWS manages backups, patches, replication, HA</td></tr> Load Balancer (ALB/NLB)</td> Nginx, HAProxy, Apache mod_proxy</code></td> Fully managed, auto-scales, integrated health checks</td></tr> Auto Scaling</td> Custom bash scripts watching top</code></td> Automatic instance scaling based on CloudWatch metrics</td></tr> IAM</td> /etc/passwd</code>, sudo</code>, LDAP/Active Directory</td> Controls AWS API access, not just OS users</td></tr> IAM Roles</td> Service accounts, systemd</code> user credentials</td> Temporary credentials that auto-rotate</td></tr> CloudWatch Metrics</td> Prometheus, Nagios, Zabbix, Grafana</td> Pre-integrated with all AWS services</td></tr> CloudWatch Logs</td> syslog</code>, journald</code>, ELK stack</td> Centralized log aggregation, SQL-like queries</td></tr> CloudWatch Alarms</td> Nagios alerts, monit</code>, custom scripts</td> Trigger actions (scaling, notifications, Lambda)</td></tr> CloudTrail</td> auditd</code>, /var/log/audit/audit.log</code></td> Every AWS API call logged (who did what, when)</td></tr> Lambda</td> Cron jobs + shell scripts</td> Event-driven functions, no server to manage</td></tr> Route 53</td> BIND, dnsmasq</code>, PowerDNS</td> Managed DNS with health checks and routing policies</td></tr> Regions</td> Multiple data center locations</td> Completely isolated, compliance/data residency boundaries</td></tr> Availability Zones</td> Separate racks/power/networking in same DC</td> Physical separation within region, low-latency links</td></tr> </tbody></table> The trade-off: You give up control (can't SSH into the Load Balancer) in exchange for less maintenance (AWS patches it for you). Whether that's worth it depends on your team size, expertise, and what you're trying to build. And if you understood everything inside the table above, you can stop here, I'm serious :\ Want to know more ? Okay</h2> Fundamentals</h3> Regions</a> are geographically isolated clusters of data centers. Each region (e.g., eu-west-3</code> for Paris, us-east-1</code> for N. Virginia) operates independently. You choose a region based on: Latency — closer to your users = faster response</li> Compliance — data residency laws may require specific locations</li> Service availability — not all services exist in all regions</li> Pricing — costs vary by region</li> </ul> Availability Zones (AZs)</a> are physically separate data centers within a region, connected by low-latency links. Deploying across multiple Zones provides fault tolerance: if one data center fails, your application survives. Organizations</a> is a service designed for central governance and management of multiple AWS accounts. They are useful for: Isolating environments (dev/staging/prod)</li> Separating billing by department or project</li> Applying security policies across accounts</li> Consolidated billing with volume discounts</li> </ul> Identity and Access Management</h3> Before you start spinning up resources, you need to understand who can do what in your AWS account. This is where IAM (Identity and Access Management)</a> comes in. IAM is AWS's authentication and authorization service. It controls who can access your AWS resources and what actions they can perform. Get this wrong, and you'll either lock yourself out or leave your infrastructure wide open to attackers. Core IAM concepts: Component</th> Purpose</th> When to Use</th></tr></thead> Users</a></td> Individual identities with permanent credentials</td> Human access via Console/CLI</td></tr> Groups</a></td> Collections of users with shared permissions</td> Organizing users by function (developers, ops, analysts)</td></tr> Roles</a></td> Temporary credentials that can be assumed</td> Services, applications, cross-account access</td></tr> Policies</a></td> JSON documents defining permissions</td> Attach to users, groups, or roles</td></tr> </tbody></table> IAM Policies</a> are JSON documents that specify allowed or denied actions. Example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"], "Resource": ["arn:aws:s3:::my-research-data/"] }] } </code></pre> This policy allows reading objects from a specific S3 bucket, nothing more. That's the principle of least privilege: grant only the minimum permissions needed to do the job. Why roles matter more than you think: The biggest mistake beginners make is hardcoding AWS credentials in their applications. I should be honest with you here, yes I've done it when I was younger. But today, please, I'm telling you, don't do this. Example scenario: Your EC2 instance needs to read files from S3. Wrong approach: Store AWS access keys in a config file on the instance. Keys can be stolen if the instance is compromised</li> Keys don't rotate automatically</li> Difficult to audit who used which credentials</li> </ul> Right approach: Attach an IAM role with S3 read permissions to the EC2 instance. No credentials stored anywhere</li> Temporary credentials rotate automatically</li> CloudTrail logs show exactly which instance accessed which resources</li> </ul> Root account protection: When you create an AWS account, you get a root user with unlimited access. This is dangerous. Best practices: Enable MFA (Multi-Factor Authentication) on the root account</li> Create IAM users for daily work, even for administrators</li> Lock away root credentials - only use them for tasks that require root (billing, account closure)</li> </ol> AWS Organizations and Service Control Policies (SCPs)</a> add another layer: they set permission boundaries across multiple accounts. Even if an IAM policy allows an action, an SCP can block it at the organization level. Useful for enforcing compliance (e.g., "no one can launch instances outside eu-west-3"). In the biomedical research context: You'd use IAM to: Give researchers read-only access to specific S3 buckets containing datasets</li> Allow the data engineering team to manage ETL pipelines (write access to certain resources)</li> Grant your analysis scripts (running on EC2/Lambda) temporary credentials to access databases</li> Prevent anyone from accidentally exposing patient data by restricting public S3 bucket creation</li> </ul> Security Groups (covered below) control network access. IAM controls identity and authorization. You need both. Network</h3> Maybe I did need this class after all. I'm having flashbacks... crimping tool... crossover cable... straight cable... cat5.... class C... IPV6... Layer 3... Layer 4.... Heeelp! \o/ A VPC (Virtual Private Cloud)</a> is your isolated network within AWS. With a VPC you define: CIDR block</a>: your IP range (e.g., 10.0.0.0/16</code> gives you 65,536 IPs)</li> Subnets</a>: subdivisions of your VPC, placed in specific Zones Public subnets: resources can have public IPs, direct internet access</li> Private subnets: no direct internet access, only internal communication</li> </ul> </li> </ul> Gateways: Internet Gateway (IGW)</a>: enables internet access for public subnets. Attach one to your VPC, add routes, done.</li> NAT Gateway</a>: lets private subnet resources reach the internet (for updates, API calls) without being directly reachable..</li> </ul> Hybrid connectivity: Site-to-Site VPN</a>: encrypted tunnel between your on-premises network and AWS VPC. Quick to set up, uses public internet, variable latency.</li> AWS Direct Connect</a>: dedicated physical connection to AWS. Consistent latency, higher bandwidth, but requires physical setup and contracts. Useful for heavy data transfer or strict latency requirements.</li> </ul> Network Security: AWS provides two complementary layers of network security: Security Groups and Network ACLs. Understanding when to use each is critical. Security Groups</a> are stateful firewalls attached to resources (EC2 instances, RDS databases, etc.). They act like bouncers at the door of individual servers. Define allowed inbound/outbound traffic by port, protocol, and source. If you allow inbound traffic on port 80, the return traffic is automatically allowed—that's what "stateful" means. Network ACLs (NACLs)</a> are stateless firewalls at the subnet level. They filter traffic entering or leaving entire subnets. Rules are evaluated in numerical order (100, 200, 300...), and the first match wins. Unlike Security Groups, NACLs require explicit rules for both inbound and outbound traffic. Security Groups vs. Network ACLs: Feature</th> Security Groups</th> Network ACLs</th></tr></thead> Scope</td> Instance level (attached to ENI)</td> Subnet level</td></tr> State</td> Stateful (return traffic auto-allowed)</td> Stateless (must allow both directions)</td></tr> Rules</td> Only Allow rules</td> Allow + Deny rules</td></tr> Evaluation</td> All rules evaluated</td> Processed in order, first match wins</td></tr> Default behavior</td> Deny all inbound, allow all outbound</td> Allow everything</td></tr> Typical use</td> Your primary security control</td> Edge cases, compliance, IP blocking</td></tr> </tbody></table> When to use what: Security Groups should be your default choice for 95% of scenarios: Allow SSH (port 22) only from your office IP</li> Let web servers accept HTTP/HTTPS from anywhere</li> Allow database connections only from application servers</li> </ul> Network ACLs are for specific edge cases: Block a known malicious IP range at the subnet boundary</li> Compliance requirements mandating subnet-level controls</li> Defense-in-depth: add a second layer of protection</li> </ul> Example scenario - Blocking an attacking IP: Your web servers (in a public subnet) are under attack from 203.0.113.0/24</code>. Security Groups can only allow traffic, not deny it. Solution: add a NACL rule. NACL Inbound Rules (evaluated in order): Rule 100: DENY TCP 80 from 203.0.113.0/24 (block attackers) Rule 200: ALLOW TCP 80 from 0.0.0.0/0 (allow everyone else) Rule : DENY ALL ALL from 0.0.0.0/0 (default deny) </code></pre> Because NACLs are stateless, you also need outbound rules for responses: NACL Outbound Rules: Rule 100: ALLOW TCP 1024-65535 to 0.0.0.0/0 (ephemeral ports) Rule *: DENY ALL ALL to 0.0.0.0/0 (default deny) </code></pre> The 1024-65535</code> range covers ephemeral ports used by client connections. Yes, this is annoying. This is why most people stick to Security Groups. Pro tip: The default NACL in your VPC allows all traffic. Most AWS users never modify it. Only touch NACLs when you have 3 specific reasons. Flow Logs</a>: Capture network traffic metadata (source/destination IP, port, protocol, action) for debugging and audit. Essential for troubleshooting connectivity issues and security investigations. Send logs to CloudWatch or S3 for analysis. Load Balancing</h3> High Availability means your application stays up even when parts of the infrastructure fail. In AWS, this typically involves deploying across multiple Availability Zones and using load balancers to distribute traffic. Elastic Load Balancing (ELB)</a> automatically distributes incoming traffic across multiple targets. AWS offers several types: Type</th> Layer</th> Best For</th></tr></thead> Application Load Balancer (ALB)</a></td> Layer 7 (HTTP/HTTPS)</td> Web apps, microservices, content-based routing</td></tr> Network Load Balancer (NLB)</a></td> Layer 4 (TCP/UDP)</td> Ultra-low latency, millions of requests/sec</td></tr> Gateway Load Balancer</a></td> Layer 3</td> Third-party virtual appliances (firewalls, IDS)</td></tr> </tbody></table> Auto Scaling</a> automatically adjusts the number of EC2 instances based on demand: Scheduled scaling: scale at predictable times (e.g., business hours)</li> Dynamic scaling: react to CloudWatch metrics (CPU, memory, custom metrics)</li> Predictive scaling: ML-based forecasting of future demand</li> </ul> Multi-AZ resilience: Deploy instances across multiple AZs, place a load balancer in front, and Auto Scaling replaces failed instances automatically. This is the foundation of most production architectures on AWS. Compute Services</h3> Compute refers to the processing power needed to run applications. AWS offers several options depending on your level of control vs. convenience: EC2 (Elastic Compute Cloud)</a> — Virtual machines you fully control. Choose your OS, instance type (CPU/RAM), and storage. Pricing models: Model</th> Description</th> Savings</th></tr></thead> On-Demand</td> Pay by the hour/second, no commitment</td> Baseline pricing</td></tr> Reserved</td> 1 or 3-year commitment for steady workloads</td> Up to 72% off</td></tr> Spot</td> Bid on unused capacity, can be interrupted</td> Up to 90% off</td></tr> Savings Plans</td> Flexible commitment across instance families</td> Up to 72% off</td></tr> </tbody></table> Container services — For containerized workloads: ECS (Elastic Container Service)</a>: AWS-native container orchestration. Simpler than Kubernetes, tightly integrated with AWS.</li> EKS (Elastic Kubernetes Service)</a>: Managed Kubernetes. Use this if you need Kubernetes compatibility or already use K8s.</li> Fargate</a>: Serverless compute for containers. No EC2 instances to manage—just define CPU/memory and run your containers.</li> </ul> When to use what: EC2: Full control needed, legacy apps, specific OS requirements</li> ECS + Fargate: Simple containerized apps, AWS-native approach</li> EKS: Kubernetes expertise on the team, multi-cloud portability needed</li> </ul> Storage</h3> AWS offers different storage types for different use cases: Amazon S3</a> — Object storage for any type of data. Highly durable (11 nines), infinitely scalable. Storage classes</a> optimize cost based on access patterns: Class</th> Access Pattern</th> Retrieval</th></tr></thead> S3 Standard</td> Frequent access</td> Instant</td></tr> S3 Intelligent-Tiering</td> Unknown/changing patterns</td> Instant (auto-moves data)</td></tr> S3 Standard-IA</td> Infrequent access</td> Instant</td></tr> S3 Glacier Instant</a></td> Archive, rare access</td> Milliseconds</td></tr> S3 Glacier Flexible</td> Archive</td> Minutes to hours</td></tr> S3 Glacier Deep Archive</td> Long-term archive</td> 12-48 hours</td></tr> </tbody></table> Lifecycle policies automatically transition objects between classes or delete them after a period. Block vs. File storage: EBS (Elastic Block Store)</a>: Block storage attached to a single EC2 instance. Think: hard drive for your VM. Great for databases.</li> EFS (Elastic File System)</a>: Shared file system mountable by multiple EC2 instances. Think: NFS. Great for shared application data.</li> </ul> Databases: Amazon RDS (Relational Database Service)</a> — Managed relational databases for PostgreSQL, (and other database providers I won't talk about because I am a free software advocate). AWS handles the operational heavy lifting: backups, patching, OS updates, and replication. You focus on schema design and queries. High Availability and scaling options: Understanding the difference between Multi-AZ and Read Replicas is critical: Feature</th> Multi-AZ</th> Read Replicas</th></tr></thead> Purpose</td> High availability (failover)</td> Read scaling, analytics</td></tr> Replication</td> Synchronous (to standby in another AZ)</td> Asynchronous (can lag slightly)</td></tr> Failover</td> Automatic (1-2 minutes)</td> Manual promotion required</td></tr> Readable</td> Standby is not accessible</td> Yes, handle read traffic</td></tr> Use case</td> Production databases requiring uptime</td> Distribute read load, reporting queries</td></tr> Cost</td> ~2x (paying for standby)</td> Additional instance cost per replica</td></tr> </tbody></table> Multi-AZ deployment: Your primary database synchronously replicates to a standby instance in a different Availability Zone. If the primary fails (hardware issue, AZ outage), RDS automatically fails over to the standby. Your application reconnects to the same endpoint—no code changes needed. This is for disaster recovery, not performance. Read Replicas: Create up to 15 read-only copies of your database. Use them to offload read traffic (analytics, reporting) from the primary. Replication is asynchronous, so there may be a slight lag (usually milliseconds to seconds). You can promote a replica to primary if needed, but it's a manual operation. Amazon Aurora</a> — AWS's proprietary database engine, PostgreSQL compatible: Performance: Up to 3x faster than PostgreSQL (according to AWS benchmarks)</li> Storage auto-scaling: Starts at 10GB, grows automatically up to 128TB in 10GB increments</li> High availability built-in: Data replicated 6 ways across 3 AZs automatically</li> Read scaling: Up to 15 Aurora Replicas with sub-10ms replica lag</li> Cost: More expensive than standard RDS, but better price-to-performance for demanding workloads</li> </ul> Aurora</a> is AWS's recommended choice for new applications that need high performance and availability. Backup Strategies: RDS provides two backup mechanisms: Automated backups</a>: Enabled by default</li> Point-in-time recovery: restore your database to any second within the retention period (1-35 days)</li> Full daily backups + transaction logs</li> Deleted when you delete the RDS instance (unless you configure a final snapshot)</li> </ul> </li> Manual snapshots</a>: User-initiated, persist until you explicitly delete them</li> Useful before major changes (schema migrations, upgrades)</li> Can be copied across regions for disaster recovery</li> </ul> </li> </ol> Pro tip: Before any risky operation (major version upgrade, schema change), create a manual snapshot. Automated backups might not cover you if the retention period expires. RDS vs. NoSQL: Not all data fits the relational model. Amazon DynamoDB</a> is AWS's managed NoSQL database (key-value and document store): Feature</th> RDS (SQL)</th> DynamoDB (NoSQL)</th></tr></thead> Data model</td> Tables with fixed schema, relations</td> Key-value / JSON documents, flexible schema</td></tr> Queries</td> Complex SQL (JOINs, aggregations)</td> Simple key lookups, limited querying</td></tr> Scaling</td> Vertical (bigger instance), Read Replicas</td> Horizontal (automatic), virtually unlimited</td></tr> Transactions</td> Full ACID support</td> Limited transactions (single-item ACID, multi-item with conditions)</td></tr> Best for</td> Complex queries, reporting, traditional apps</td> High-throughput simple queries, IoT, gaming, mobile backends</td></tr> Pricing</td> Pay for instance size (even if idle)</td> Pay for storage + read/write capacity used</td></tr> </tbody></table> When to use what: RDS: You need SQL, complex queries, transactions, existing relational schema, or you're migrating from on-premises MySQL/PostgreSQL</li> DynamoDB: Key-value access patterns, need massive scale, unpredictable traffic spikes, single-digit millisecond latency requirements</li> </ul> Most traditional applications (e-commerce, ERP, content management) use RDS. DynamoDB shines for high-scale, low-latency use cases where you don't need complex querying. Observability</h3> Observability = understanding what's happening inside your systems. AWS provides Amazon CloudWatch</a> as the central service for this. Metrics</a>: Numerical data points over time. AWS services automatically send metrics (CPU usage, network traffic, request counts). You can also publish custom metrics from your applications. Logs</a>: Centralized log storage and analysis. Stream logs from EC2, Lambda, containers, or any application. Use Log Insights to query across log groups with a SQL-like syntax. Alarms</a>: Watch metrics and trigger actions when thresholds are breached: Send notifications via SNS (email, SMS, Slack)</li> Trigger Auto Scaling to add/remove instances</li> Execute Lambda functions for custom remediation</li> </ul> Example alarm: "If average CPU > 80% for 5 minutes, add 2 instances and notify the ops team." Beyond CloudWatch: AWS X-Ray</a>: Distributed tracing for microservices</li> CloudTrail</a>: Audit log of all API calls in your account</li> </ul> One picture is worth thousand words</h2>

The Container image</h2>

The VPS provisioning</h2>

SSL & Let's Encrypt</h2>

Operations</h2>

Vectors

Going further</h2>
Once you've internalized the two-lens model, here's where to take it next:</p>

AWS (Day 1)

One picture is worth thousand words</h2>

Nskm

From rsync to CI/CD: automating a static site deployment

The journey</h2>

GitLab CI pipeline</h2>
TODO: gitlab ci file ???</p>

Nskm

From rsync to CI/CD: automating a static site deployment

The journey</h2>

GitLab CI pipeline</h2> TODO: gitlab ci file ???</p>

The Container image</h2>

The VPS provisioning</h2>

SSL & Let's Encrypt</h2>

Operations</h2>

Vectors

Going further</h2> Once you've internalized the two-lens model, here's where to take it next:</p>

AWS (Day 1)

GitLab CI pipeline</h2>
TODO: gitlab ci file ???</p>

Going further</h2>
Once you've internalized the two-lens model, here's where to take it next:</p>