Nskm

C Pointers (WIP)

2024-01-20T00:00:00+00:00

In my last</a> post, I've written about lots of things but not about C pointers. I tought the topic deserves a post in itself because it's one of the most important feature of the C language.

Let's start with some music</a> please.

ToC</h2>

Introduction</a></li>
Declaration & initialization</a></li>
Operators</a></li>
Use cases</a>

Arrays & strings manipulation</a></li>
Pass by reference</a></li>
Memory allocation</a></li>
Data structure implementation</a></li>
Functions pointers</a></li>
Hardware interface</a></li> </ul> </li>
Going further</a></li> </ul>
Introduction:</h2>
A pointer is a variable that contains the memory address of another variable. A pointer allows us to indirectly access and manipulate the value of the variable that is pointed to. Like other variables, C pointers have a type.

Source</a> </center>
Declaration and initialization:</h2>
#include <stdio.h> #include <stdlib.h> int main () { int foo = 42; int bar = 83; int ptr = &foo; // Pointer variable of type int int **ptrptr = &ptr; // pointer to a pointer printf("Value of foo: %d\n", foo); printf("Value of ptr is: %p, and ptr is pointing to: %d\n", (void )ptr, ptr); printf("Value of ptrptr is: %p & ptrptr is pointing to: %p\n", (void )ptrptr, (void )ptr); printf("\n"); *ptr = 12; printf("Value of foo after modification via ptr: %d\n", foo); **ptrptr = 13; printf("Value of foo after modification via ptr to ptr: %d\n", foo); printf("\n"); ptr = &bar; printf("Value of ptr is now: %p, and ptr is now pointing to: %d\n", (void )ptr, ptr); printf("Value of ptrptr, still: %p & ptrptr is now pointing to: %p\n", (void )ptrptr, (void )ptr); exit(EXIT_SUCCESS); } </code></pre> And the output of the previous program is:
Value of foo: 42 Value of ptr is: 0x7ffc6155fd60, and ptr is pointing to: 42 Value of ptrptr is: 0x7ffc6155fd68 & ptrptr is pointing to: 0x7ffc6155fd60 Value of foo after modification via ptr: 12 Value of foo after modification via ptr to ptr: 13 Value of ptr is now: 0x7ffc6155fd64, and ptr is now pointing to: 83 Value of ptrptr, still: 0x7ffc6155fd68 & ptrptr is now pointing to: 0x7ffc6155fd64 </code></pre>
C pointer operators:</h2>
C offers operators for pointer manipulation:

Address-of operator (&): to return the memory address of a variable</li>
Dereference operator (): to access the value stored at the memory address pointed to</li>
Arithmetic operators: to perform arithmetic operations on C pointers, addition (+), subtraction (-), increment (++), and decrement (--)</li> </ul>
Use cases for C pointers:</h2>
All modern programming languages have pointers in some form or other. Pointers are used everywhere there is a program, even when we don't see them. Let's explore some of those use cases.
Manipulating array and string elements:</h3>
C Pointers can be used to access and manipulate elements in arrays and strings. C Pointers provide a way to efficiently iterate over arrays and perform operations on each item of the sequence.
int arr[5] = {1, 2, 3, 4, 5}; int aptr = arr; // print items of the sequence using array indices for (int i = 0; i < 5; i++) { printf("%d ", arr[i]); } printf("\n"); // print elements of the array using pointer for (int i = 0; i < 5; i++) { printf("%d ", aptr); aptr++; } // print items of the sequence using pointer arithmetic // and the fact that arr is actually a pointer to the first item of the arr for (int i = 0; i < 5; i++) { printf("%d ", arr+i); } </code></pre> // an array of characters char str[] = "Hello world from West Africa"; char sptr = str; // a pointer to the first element of the array int i = 0; while(i < 28){ // while i is less than 28, print the character at i-th index printf("%c ", str[i]); // increment i to go to the next index i++; } printf("\n"); // while the value that is pointed to is different from the "\0" character while (sptr != '\0') { // print the char printf("%c ", sptr); // increment pointer (to go to the next character) sptr = sptr + 1; } printf("\n"); int m = 0; while(m < 28){ // str is actually a pointer to the first item of the object // so str+m is the address of the m-ieth item of the object printf("%c ", (str+m)); m++; } </code></pre> // array of pointers to characters char *names[] = {"Alice", "Bob", "Charlie", "David", "Emma"}; // Pointer to the first element of the first element of the array of strings char **ptr_names = names; for (int j = 0; j < 5; j++) { // Print elements of the array of strings using indices printf("%s\n", names[j]); } for (int x = 0; x < 5; x++) { // Print the element currently pointed to printf("%s\n", *ptr_names); ptr_names = ptr_names + 1; } </code></pre> Passing arguments by reference:</h3> C Pointers can be used to pass arguments by reference to functions, allowing the function to modify the value of the original variable. This is particularly useful in the following situations: working with large data structures </li> you need to modify the value of a variable in a function</li> you need to optimize code by reducing the number of copies of data that need to be made</li> you need to return more than one value</li> </ul> Examples: </code></pre> Allocating memory:</h3> C Pointers are also used to allocate memory dynamically at run-time using functions like malloc()</a>, calloc()</a>, and realloc()</a>. Examples: </code></pre> Implementing data structures:</h3> Linked lists</a>, Stacks</a>, Queues</a>, Trees</a>, Graphs</a> and many other data structures are implemented using C Pointers. Examples: </code></pre> Calling functions:</h3> A function pointer, is a pointer which point to the memory address of a function instead of pointing to the address of a data object. They are used to: store functions in data structures</li> pass functions as arguments to other functions</li> call functions dynamically at runtime</li> </ul> Here are three examples of using function pointers in C: Interfacing with hardware:</h3> C Pointers are often used in low-level programming to interact with hardware devices and memory-mapped registers. They provide a way to access specific memory addresses and manipulate hardware directly. Going further:</h2> To learn more about C pointers, we recommend the following pointers: 0x7ffc6155fd68</a></li> 0x7ffc6155fd69</a></li> 0x7ffc6155fd70</a></li> 0x7ffc6155fd71</a></li> </ul>

C 2023-09-25T00:00:00+00:00 I spent some time adding type annotations</a> to some Python code. Shoutout to all the people at Logilab</a>, a really awesome team doing really awesome things. All those type annotations made me more and more interested in static typing... and in C programming. Target audience: people who already have an experience with programming. People who may have written some C programs in the past and who want to refresh their memory. Context: I'm working on: I'm working on Linux 5.13.0-52-generic #59-Ubuntu SMP Wed Jun 15 20:17:13 UTC 2022 x86_64</code> </li> And I'm using GCC (Ubuntu 11.2.0-7ubuntu2) 11.2.0</code></li> </ul> Music please</a> ToC</h2> Setup</a></li> Primitive types</a> integers</a></li> floats</a></li> booleans</a></li> void</a></li> </ul> </li> Composite types</a> structs</a></li> enums</a></li> unions</a></li> arrays</a></li> strings</a></li> </ul> </li> Constants</a></li> </ul> The setup:</h2> To write C programs, we need: An operating system</a> that respect your freedom and privacy.</li> A C compiler such as gcc</a> or clang</a>.</li> A text editor</a> that respects</a> your freedom and privacy.</li> (Optional) a Bash script to generate the project boilerplate</a>.</li> (Optional) a Makefile</a> to format, build, install and run the program quickly.</li> </ul> Types:</h2> Variable, a named location in the computer's memory where data can be stored and accessed. Type, define the size, the format, the behavior, the operations that can be performed on the variable. Let's talk about the common types in C programming. Primitive types:</h2> Primitive data types, or "basic data types" or "fundamental data types" are the most basic data types that can be used for representing simple values such as numbers, characters, etc... The primitive data types are... Integer types</h3> char: represents a single character (usually 1 byte)</li> short: represents a small integer</li> int: represents an integer</li> long: represents a long integer</li> long long: represents a very long integer</li> unsigned short: represent a small positive integer</li> unsigned int: represents a positive integer</li> unsigned long: represents a long positive integer</li> unsigned long long: represents a very long positive integer</li> </ul> This list is not exhaustive, there is also what we call fixed width integer types and what we call minimum width integer types. We won't talk about them here, please browse this link</a> for reference. To find the range of a particular type, we can look for the limits.h</code> file online</a>. Or, we can also look for the same file inside our computer: lim (trunk *%) >> find /usr -type f -name "limits.h" /usr/src/linux-headers-5.13.0-51/include/linux/limits.h /usr/src/linux-headers-5.13.0-51/include/uapi/linux/limits.h /usr/src/linux-headers-5.13.0-51/include/vdso/limits.h /usr/src/linux-headers-5.13.0-52/include/linux/limits.h /usr/src/linux-headers-5.13.0-52/include/uapi/linux/limits.h /usr/src/linux-headers-5.13.0-52/include/vdso/limits.h /usr/include/linux/limits.h /usr/include/c++/11/tr1/limits.h /usr/include/limits.h /usr/lib/gcc/x86_64-linux-gnu/11/include/limits.h /usr/lib/x86_64-linux-gnu/perl5/5.32/Tk/pTk/compat/limits.h /usr/lib/llvm-13/lib/clang/13.0.0/include/limits.h lim (trunk *%) >> </code></pre> If we open, let's say, the file /usr/include/limits.h</code>, we can see interesting informations like : # define CHAR_MAX UCHAR_MAX # else # define CHAR_MIN SCHAR_MIN # define CHAR_MAX SCHAR_MAX # endif /* Minimum and maximum values a `signed short int' can hold. */ # define SHRT_MIN (-32768) # define SHRT_MAX 32767 /* Maximum value an `unsigned short int' can hold. (Minimum is 0.) */ # define USHRT_MAX 65535 /* Minimum and maximum values a `signed int' can hold. */ # define INT_MIN (-INT_MAX - 1) # define INT_MAX 2147483647 /* Maximum value an `unsigned int' can hold. (Minimum is 0.) */ # define UINT_MAX 4294967295U /* Minimum and maximum values a `signed long int' can hold. */ # if __WORDSIZE == 64 # define LONG_MAX 9223372036854775807L # else # define LONG_MAX 2147483647L </code></pre> Floating-point types:</h3> Used to represent real numbers with decimal points. More informations here</a> float: represents a single-precision floating-point number</li> double: represents a double-precision floating-point number</li> long double: represents an extended-precision floating-point number</li> </ul> Here is an example showing how to use float values: #include <stdio.h> #include <stdlib.h> int main() { char input[100]; float radius, area; printf("Enter the radius of the circle: "); if(fgets(input, sizeof input, stdin) != NULL) { radius = strtof(input, NULL); area = 3.14 * radius * radius; printf("Area of the circle: %.2f\n", area); } exit(EXIT_SUCCESS); } </code></pre> I have found the file containing all the constants definitions for float values, but I am not (yet) able to correctly understand what's going on in this header file: lim (trunk *%) >> find /usr -type f -name "float.h" /usr/include/tcl8.6/tcl-private/compat/float.h /usr/include/c++/11/tr1/float.h /usr/lib/gcc/x86_64-linux-gnu/11/include/float.h /usr/lib/llvm-13/lib/clang/13.0.0/include/float.h lim (trunk *%) >> </code></pre> Here is a snippet of things that may be of interest for anyone looking for the ranges: #if defined __STDC_VERSION__ && __STDC_VERSION__ > 201710L /* Maximum finite positive value with MANT_DIG digits in the significand taking their maximum value. */ #undef FLT_NORM_MAX #undef DBL_NORM_MAX #undef LDBL_NORM_MAX #define FLT_NORM_MAX __FLT_NORM_MAX__ #define DBL_NORM_MAX __DBL_NORM_MAX__ #define LDBL_NORM_MAX __LDBL_NORM_MAX__ /* Whether each type matches an IEC 60559 format (1 for format, 2 for format and operations). */ #undef FLT_IS_IEC_60559 #undef DBL_IS_IEC_60559 #undef LDBL_IS_IEC_60559 #define FLT_IS_IEC_60559 __FLT_IS_IEC_60559__ #define DBL_IS_IEC_60559 __DBL_IS_IEC_60559__ #define LDBL_IS_IEC_60559 __LDBL_IS_IEC_60559__ </code></pre> Boolean type:</h3> The data type that can have two possible values: true or false. Include the "stdbool.h"</a> header file in your program to use this data type. Example: #include <stdio.h> #include <stdlib.h> #include <stdbool.h> #include <string.h> bool strtobool(const char* str) { if (strcmp(str, "true") == 0 || strcmp(str, "1") == 0) { return true; } else if (strcmp(str, "false") == 0 || strcmp(str, "0") == 0) { return false; } else { fprintf(stderr, "Invalid input: %s\n", str); exit(EXIT_FAILURE); } } int main() { char name[100]; char answer[5]; bool is_learning; printf("Enter your name: "); if(fgets(name, sizeof name, stdin) != NULL) { printf("Your name is %s\n", name); } printf("Are you learning C programming: "); if(fgets(answer, sizeof answer, stdin) != NULL) { is_learning = strtobool(answer); if (is_learning) printf("Very good, keep learning.\n"); else printf("You should learn C programming.\n"); } exit(EXIT_SUCCESS); } </code></pre> Void type:</h3> Void is a keyword to use as a placeholder where you would put a data type, to represent "no data". The void data type represents the absence of type. Void types are used in the following situations: to state that the specific type of the variable is not known yet (void pointers). </li> to state that the function returns no result (void functions).</li> to state that the function has no parameters.</li> </ul> IMPORTANT: We can't directly dereference void pointers because the compiler doesn't know the size or type of data being pointed to. We need to explicitly cast them to the correct type before dereferencing. </blockquote> Here's an example to illustrate the usage of void type: #include <stdio.h> #include <stdlib.h> // greet is a function that // returns no result // and takes an unkown/undefined list of arguments void greet(){ printf("Hello world\n"); } void main(){ // notice how we can pass all the arguments we want greet("foo", 3.14, true, 1); exit(EXIT_SUCCESS); } </code></pre> A function declared with the void data type can't return a value, we cannot use the return keyword inside a void function: cscripts >> gcc main.c && ./a.out main.c: In function ‘greet’: main.c:57:10: warning: ‘return’ with a value, in function returning void 57 | return 5432; | ^~~~ main.c:56:6: note: declared here 56 | void greet(void){ | ^~~~~~ </code></pre> Another example: #include <stdio.h> #include <stdlib.h> // greet is a function that // returns an int // and takes no arguments int greet(void){ return 5432; } void main(){ // Because greet returns a result, I can retrieve and print it printf("Greet returned the value: %d\n", greet()); exit(EXIT_SUCCESS); } </code></pre> If we try to pass an argument to the greet function, we will receive an error message: cscripts >> gcc main.c && ./a.out main.c: In function ‘main’: main.c:57:44: error: too many arguments to function ‘greet’ 57 | printf("Greet returned the value: %d\n", greet("foo")); | ^~~~~ main.c:52:5: note: declared here 52 | int greet(void){ | ^~~~~ </code></pre> Last example: #include <stdio.h> #include <stdlib.h> void main(){ int age = 5432; char* name = "Nsukami_"; // foo is a pointer that point to a not yet knwon data type void* foo; // let's point to an integer foo = &age; // explicit cast to the correct type before dereferencing printf("age is %d\n", *((int*)foo)); // let's point to something else, for ex, a string foo = &name; // explicit cast to the correct type before dereferencing printf("Name is %s\n", *((char**)foo)); exit(EXIT_SUCCESS); } </code></pre> Composite types</h2> Any data type which can be constructed using the language's primitive data types and other composite types. Structures:</h3> Structs (short for structure), a user-defined data type that allows us to group together related data items of different types. Those data items are known as members or fields. #include <stdio.h> #include <stdlib.h> // An address is something with 2 attributes // a city street AND an house number typedef struct address { char citystreet[100]; int housenumber; } Address; void main() { Address my_address = { "Planet 42, Rue de la Liberté", 5432, }; printf("I live in %s. My house number is: %u\n", my_address.citystreet, my_address.housenumber); // how to update one of the fields my_address.housenumber = 443; printf("I live in %s. My house number is now: %u\n", my_address.citystreet, my_address.housenumber); } </code></pre> Enum data type:</h3> An enumeration is a special kind of data type defined by the user. An enumeration consists of a set of constant integers that are named by the user. #include <stdio.h> typedef enum Level { LOW = 1, MEDIUM, HIGH } Level; int main() { Level level = MEDIUM; switch (level) { case 1: printf("Low level"); break; case 2: printf("Medium level"); break; case 3: printf("High level"); break; default: printf("Unkown level value"); } exit(EXIT_SUCCESS); } </code></pre> We said earlier an enumeration is a special kind of data type because. </code></pre> Unions:</h3> An union is a user-defined data type that allows different data types to be stored in the same memory location. It enables us to create a variable that can hold different types of data, but only one type at a time. #include <stdio.h> // Struct that combines an enum and a union typedef struct { // The type of our object could be either INT, either FLOAT, either STRING enum { INT, FLOAT, STRING } type; // all 3 values share the same memory location, only one can be set // value can be a int OR a float OR a string union { int intValue; float floatValue; char stringValue[20]; } value; } Object; // Function to print the object based on its type void print(Object o) { // check the type of our object switch(o.type) { case INT: // then access the correct value printf("Integer: %d\n", o.value.intValue); break; case FLOAT: printf("Float: %.2f\n", o.value.floatValue); break; case STRING: printf("String: %s\n", o.value.stringValue); break; } } int main() { // Creating objects of different types Object o1, o2, o3; o1.type = INT; o1.value.intValue = 10; o2.type = FLOAT; o2.value.floatValue = 3.14; o3.type = STRING; strcpy(o3.value.stringValue, "Hello"); // Printing the data objects print(o1); print(o2); print(o3); exit(EXIT_SUCCESS); } </code></pre> Arrays:</h3> A type consisting of nonempty items of the same nature stored using contiguous memory location. The number of items (the size) never changes during the array lifetime. Example: #include <stdio.h> #include <stdlib.h> int main() { float numbers[] = {2.5, 3.8, 4.2, 1.9, 5.6}; int size = sizeof(numbers) / sizeof(numbers[0]); float sum = 0.0, average; for (int i = 0; i < size; i++) { sum += numbers[i]; } average = sum / size; printf("Average: %.2f\n", average); exit(EXIT_SUCCESS); } </code></pre> Strings:</h3> A string is just a sequence of characters stored inside an array and ending with the null character: '\0'. We don't have strings as we have them in Python programming. #include <stdio.h> #include <stdlib.h> int main() { char name[100]; printf("Enter your name: "); if(fgets(name, sizeof name, stdin) != NULL) { printf("Your name is %s\n", name); } exit(EXIT_SUCCESS); } </code></pre> To know what are the functions available for string manipulation, Python developers will type help(str)</code>: Help on class str in module builtins: class str(object) | str(object='') -> str | str(bytes_or_buffer[, encoding[, errors]]) -> str | | Create a new string object from the given object. If encoding or | errors is specified, then the object must expose a data buffer | that will be decoded using the given encoding and error handler. | Otherwise, returns the result of object.__str__() (if defined) | or repr(object). | encoding defaults to sys.getdefaultencoding(). | errors defaults to 'strict'. | | Methods defined here: | | __add__(self, value, /) | Return self+value. | | __contains__(self, key, /) | Return bool(key in self). | | __eq__(self, value, /) | Return self==value. | | __format__(self, format_spec, /) : </code></pre> C developers will look inside the <string.h></code> header file</a>. nskm2 >> find /usr/include -type f -name "string.h" /usr/include/linux/string.h /usr/include/tcl8.6/tcl-private/compat/string.h /usr/include/string.h nskm2 >> </code></pre> They can also read the man pages, man string</code>: STRING(3) Linux Programmer's Manual STRING(3) NAME stpcpy, strcasecmp, strcat, strchr, strcmp, strcoll, strcpy, strcspn, strdup, strfry, strlen, strncat, strncmp, strncpy, strncasecmp, strpbrk, strrchr, strsep, strspn, strstr, strtok, strxfrm, index, rindex - string operations SYNOPSIS #include <strings.h> int strcasecmp(const char *s1, const char *s2); Compare the strings s1 and s2 ignoring case. int strncasecmp(const char *s1, const char *s2, size_t n); Compare the first n bytes of the strings s1 and s2 ignoring case. char *index(const char *s, int c); Return a pointer to the first occurrence of the character c in the string s. char *rindex(const char *s, int c); Return a pointer to the last occurrence of the character c in the string s. #include <string.h> char *stpcpy(char *dest, const char *src); Copy a string from src to dest, returning a pointer to the end of the resulting string at dest. Manual page string(3) line 1 (press h for help or q to quit) </code></pre> Constants:</h2> A fixed value that cannot be, that should not be changed during the execution of a program. If we have a fixed value that is used in calculations, we can define it as a constant to avoid hard-coding the value multiple times.</li> </ol> // let's define PI as a constant const int PI = 3.14159; int radius = 5; // let's reuse our constant int area = PI * radius * radius; </code></pre> Trying to modify our constant will give us an error message: main.c: In function ‘main’: main.c:129:6: error: assignment of read-only variable ‘PI’ 129 | PI = 4.5; | ^ </code></pre> Another way to use constants is this one. Instead of using arbitrary numbers directly in your code, you can define them as constants with meaningful names to improve code readability and maintainability.</li> </ol> const int LENGTH = 3; const int MIN_AGE = 18; const int MAX_AGE = 50; char input[LENGTH]; int age; printf("Enter your age: "); if(fgets(input, sizeof input, stdin) != NULL) { age = strtol(input, NULL, 10); if (age < MIN_AGE || age > MAX_AGE) { printf("%d is an invalid age! Please enter an age between %d and %d.\n", age, MIN_AGE, MAX_AGE); } } </code></pre> When declaring an array, we can use a constant to specify its size, making it easier to modify if needed.</li> </ol> // In this example, `ARRAY_SIZE` is declared as a constant to specify the size of an array. By using a // constant, you can easily change the size of the array by modifying the constant's value in a single place. const int ARRAY_SIZE = 10; int arr[ARRAY_SIZE]; </code></pre> We can also declare function parameters as const</code> to ensure that the function does not modify the parameter:</li> </ol> // This helps prevent accidental modifications and indicates that the function is read-only. void print(const int arr[], int size) { for (int i = 0; i < size; i++) { printf("%d ", arr[i]); } } </code></pre> Conclusion:</h2> I really hope you've learned something. I did \o/ In the next episode, I will try to write about header files or pointers or something else depending on the current moon, the weather and the stars alignment. Also, thanks to Orbifx</a> for taking the time to answer all my questions. More on the topic:</h1> Modern C, Jens Gustedt</a></li> The GNU C Reference Manual</a></li> Explanation about how to write a Makefile</a></li> Make false and true first-class language features v4 proposal for C23</a></li> </ul> On a completely differnt note:</h1> Des organismes aux algorithmes, l'odyssée computationnelle du vivant | Xavier Berthet | TEDxINPENSAT</a></li> The first DjangoCon Africa, in Zanzibar, Tanzania.</a></li> </ul> Decorators 2023-02-06T00:00:00+00:00 I was watching a really interesting video</a> about buildings architecture in New York, when one of my DIT</a> student asked me to explain to him what was happening in this piece of code</a>. A function I have written few months ago to check for valid authentication tokens. In this post, I will try to show how decorators in Python</a>, a powerful concept, allows us to apply new functionality without modifying the existing structure of an object. Music please</a>. TOC:</h1> 1 A function doing one thing</a></li> 2 A decorator, modifying the behaviour of the function</a></li> 3 A decorator should keep important informations</a></li> 4 A decorator should receive a flexible number of argument</a></li> 5 A decorator can also have parameters</a></li> 6 We are still able to access the decorated function</a></li> 7 An interesting example</a></li> 8 Even more interesting, how decorators are used elsewhere</a></li> </ul> 1. A function:</h1> Let's suppose we want to join 2 paths: In [2]: from pathlib import Path In [3]: a = Path("foo") In [4]: a Out[4]: PosixPath('foo') In [5]: b = Path("bar") In [6]: a.joinpath(b) Out[6]: PosixPath('foo/bar') In [7]: </code></pre> Let's suppose we usually combine 2 paths together. So much that we created a function for that purpose: In [21]: path_t = str | Path In [22]: def jp(path1: path_t, path2: path_t) -> path_t: ...: """Join 2 paths together""" ...: if isinstance(path1, str): ...: path1 = Path(path1) ...: return path1.joinpath(path2) ...: In [23]: jp(b, a) Out[23]: PosixPath('bar/foo') In [24]: jp(a, b) Out[24]: PosixPath('foo/bar') In [25]: jp("corge", a) Out[25]: PosixPath('corge/foo') In [26]: jp("corge", "qux") Out[26]: PosixPath('corge/qux') </code></pre> 2. A decorator modifying the behaviour of the function:</h1> While it may seem confusing at first, a decorator is really just a function that takes a function and returns a function. Let me explain. Suppose we usually combine our home path with another path. Let's suppose the first parameter is often our home path. Let's suppose we can't modify the original function. Python functions are first-class citizens</a>. That means: We can pass a function as an argument of another function.</li> We can assign a function to a variable.</li> We can return a function as a value from another function. </li> </ul> In [38]: def homeprefix(func: Callable[[path_t, path_t], path_t]) -> Callable[[path_t], path_t]: ...: def wrapper(m: path_t) -> path_t: ...: """A wrapper that takes one parameter.""" ...: return func("/home/nsukami", m) ...: return wrapper ...: In [39]: f = homeprefix(jp) In [40]: f("bar") Out[40]: PosixPath('/home/nsukami/bar') In [41]: f(b) Out[41]: PosixPath('/home/nsukami/bar') In [42]: f(a) Out[42]: PosixPath('/home/nsukami/foo') In [43]: f("corge/qux") Out[43]: PosixPath('/home/nsukami/corge/qux') In [44]: f(Path("spam")) Out[44]: PosixPath('/home/nsukami/spam') In [45]: </code></pre> As strange as it may seems, we could have done the following: In [45]: jp = homeprefix(jp) # there is an alternative syntax In [46]: jp(a) Out[46]: PosixPath('/home/nsukami/foo') In [47]: jp(b) Out[47]: PosixPath('/home/nsukami/bar') In [48]: jp("foobar") Out[48]: PosixPath('/home/nsukami/foobar') In [49]: jp(Path("corge")) Out[49]: PosixPath('/home/nsukami/corge') In [50]: </code></pre> Instead of jp = homeprefix(jp)</code> we could use the following syntax which better vehicle our intent: modifying the behaviour of an existing function. Simply put: decorating our original function: In [50]: def homeprefix(func): ...: def wrapper(m): ...: """A function that takes one parameter.""" ...: return func("/home/nsukami", m) ...: return wrapper ...: In [51]: @homeprefix # here we are, decorating jp function ...: def jp(path1: path_t, path2: path_t) -> path_t: ...: """Join 2 paths together""" ...: if isinstance(path1, str): ...: path1 = Path(path1) ...: return path1.joinpath(path2) ...: In [52]: jp(Path("corge")) Out[52]: PosixPath('/home/nsukami/corge') In [53]: jp("corge/qux") Out[53]: PosixPath('/home/nsukami/corge/qux') In [54]: jp(a) Out[54]: PosixPath('/home/nsukami/foo') In [55]: jp(b) Out[55]: PosixPath('/home/nsukami/bar') In [56]: </code></pre> 3. A decorator should keep important informations:</h1> Now we have another problem. Because we are literally replacing one function with another, we're also losing important informations like the docstring and the name: In [61]: ?jp Signature: jp(m) Docstring: A function that takes one parameter. File: ~/GIT/nskm2/<ipython-input-59-b4f67b1e800d> Type: function In [62]: jp.__name__ Out[62]: 'wrapper' In [63]: </code></pre> That's the reason we have functools.wraps</a>. Wraps</code> takes the decorated function and adds the functionality of copying over the function name, docstring, arguments list, etc to the returned function. And since wraps is itself a decorator: In [63]: from functools import wraps In [64]: def homeprefix(func): ...: @wraps(func) ...: def wrapper(m): ...: """A function that takes one parameter.""" ...: return func("/home/nsukami", m) ...: return wrapper ...: In [65]: @homeprefix ...: def jp(path1: path_t, path2: path_t) -> path_t: ...: """A function that join 2 paths together.""" ...: if isinstance(path1, str): ...: path1 = Path(path1) ...: return path1.joinpath(path2) ...: In [66]: ?jp Signature: jp(path1: str | pathlib.Path, path2: str | pathlib.Path) -> str | pathlib.Path Docstring: A function that join 2 paths together. File: ~/GIT/nskm2/<ipython-input-65-1703cd8a40dc> Type: function In [67]: jp.__name__ Out[67]: 'jp' In [68]: </code></pre> 4. A decorator should receive a flexible number of argument:</h1> Now, let's suppose our original function can take many arguments: In [101]: def jp(*args: path_t) -> path_t: ...: """A function that join many paths together.""" ...: if args: ...: path1 = Path(args[0]) ...: return path1.joinpath(*args[1:]) ...: else: ...: raise ValueError("args should be longer") ...: In [102]: jp("bac", "abc") Out[102]: PosixPath('bac/abc') In [103]: jp("bac") Out[103]: PosixPath('bac') In [104]: jp(Path("abc"), Path("foo"), "bar") Out[104]: PosixPath('abc/foo/bar') </code></pre> The previous decorator is not flexible enough because the returned function can't take more than 1 parameter: In [106]: def homeprefix(func): ...: @wraps(func) ...: def wrapper(m): ...: """A function that takes one parameter.""" ...: return func("/home/nsukami", m) ...: return wrapper ...: In [107]: @homeprefix ...: def jp(*args: path_t) -> path_t: ...: """A function that join many paths together.""" ...: if args: ...: path1 = Path(args[0]) ...: return path1.joinpath(*args[1:]) ...: else: ...: raise ValueError("args should be longer") ...: ...: In [108]: jp("a") Out[108]: PosixPath('/home/nsukami/a') In [109]: jp("a", "b") --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In [109], line 1 ----> 1 jp("a", "b") TypeError: jp() takes 1 positional argument but 2 were given In [110]: </code></pre> Let's rewrite our decorator so that the returned function can take an infinte number or arguments: In [113]: def homeprefix(func): ...: @wraps(func) ...: def wrapper(*args): # here is our update: using *args ...: return func("/home/nsukami", *args) ...: return wrapper ...: In [114]: @homeprefix ...: def jp(*args: path_t) -> path_t: ...: """A function that join many paths together.""" ...: if args: ...: path1 = Path(args[0]) ...: return path1.joinpath(*args[1:]) ...: else: ...: raise ValueError("args should be longer") ...: In [115]: jp() Out[115]: PosixPath('/home/nsukami') In [116]: jp("b", "c", "d") Out[116]: PosixPath('/home/nsukami/b/c/d') In [117]: jp("b", "c", "d", Path("foo")) Out[117]: PosixPath('/home/nsukami/b/c/d/foo') In [118]: jp(Path("foo"), Path("bar")) Out[118]: PosixPath('/home/nsukami/foo/bar') In [119]: </code></pre> With the help of the special parameters *args and **kwargs, decorators become more generic. That way if we need the same kind of decorator in different functions with different arguments, we can write just one decorator. In general, a decorator is written like this: def decorator(func): @wraps(func) def wrapper(*args, **kwargs): func(*args, **kwargs) return wrapper </code></pre> 5. A decorator can also have parameters:</h1> Let's suppose people using the decorator want to be able to set what should the prefix used as the first parameter of our jp function. Let's suppose we want the decorator to receive one parameter, which is the prefix: In [70]: def homeprefix(prefix): # this function builds and returns a decorator ...: def outer(func): # this function is the decorator ...: @wraps(func) ...: def inner(*args, **kwargs): # the function wrapping the original function ...: """A function that takes one parameter.""" ...: return func(prefix, *args, **kwargs) ...: return inner ...: return outer ...: In [71]: @homeprefix("/home/nsukami") ...: def jp(path1: path_t, path2: path_t) -> path_t: ...: """A function that join 2 paths together.""" ...: if isinstance(path1, str): ...: path1 = Path(path1) ...: return path1.joinpath(path2) ...: In [72]: jp(a) Out[72]: PosixPath('/home/nsukami/foo') In [73]: jp(b) Out[73]: PosixPath('/home/nsukami/bar') In [74]: jp("corge") Out[74]: PosixPath('/home/nsukami/corge') In [75]: </code></pre> We understand better what's going on when we do this instead: In [75]: decorator = homeprefix("/home/patrick") # setup a prefix In [76]: @decorator # decorate ...: def jp(path1: path_t, path2: path_t) -> path_t: ...: """A function that join 2 paths together.""" ...: if isinstance(path1, str): ...: path1 = Path(path1) ...: return path1.joinpath(path2) ...: In [77]: jp("corge") Out[77]: PosixPath('/home/patrick/corge') In [78]: jp("quux") Out[78]: PosixPath('/home/patrick/quux') In [79]: </code></pre> 6. We are still able to access the decorated function:</h1> Let's suppose for some reason we want to test the function as if it was not decorated. Let's suppose we still want to use the underlying function. To achieve that, we'll use the __wrapped__</code> attribute (assuming we used the functools.wrap</code> function). Somewhere</a> in the documentation, we can read: The original underlying function is accessible through the wrapped attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache. </blockquote> In [127]: jp("foo") Out[127]: PosixPath('/home/nsukami/foo') In [128]: jp.__wrapped__("foo") # original function Out[128]: PosixPath('foo') In [129]: </code></pre> 7. An interesting example:</h1> #!/usr/bin/env python3 import csv import sqlite3 import pathlib from functools import wraps here = pathlib.Path(__file__) db = here.parent.parent.parent / "sqlite_stuff" / "chinook" / "chinook.db" def connect(db): """A decorator to connect to an SQLite database and execute a query.""" conn = sqlite3.connect(db) conn.isolation_level = None def outer(f): @wraps(f) def inner(*args, **kwargs): query = f(*args, **kwargs) rset = conn.execute(query).fetchall() return rset return inner return outer def columns(*columns, limit=10): """A decorator to specify what columns should be retrieved.""" cols = ", ".join(columns) def outer(f): @wraps(f) def inner(*args, **kwargs): query = f(*args, **kwargs).split("*") start, end = query[0].strip(), query[1].strip() return f"{start} {cols} {end} limit {limit}" return inner return outer def as_csv(where="./result.csv"): """A decorator to save the result as a csv file.""" def outer(f): @wraps(f) def inner(*args, **kwargs): rset = f(*args, **kwargs) with open(where, "w", newline="") as dest: csv_writer = csv.writer( dest, delimiter=",", quoting=csv.QUOTE_ALL, lineterminator=";\r\n" ) csv_writer.writerows(rset) return len(rset) return inner return outer ### @as_csv() @connect(db) @columns("title", "artistId") def run(table="albums"): query = f"select * from {table}" return query if __name__ == "__main__": print(run()) </code></pre> 8. Even more interesting:</h1> import sqlite3 import pathlib from functools import wraps here = pathlib.Path(__file__) db = here.parent.parent.parent / "sqlite_stuff" / "chinook" / "chinook.db" def connect(cls): conn = sqlite3.connect(db) conn.isolation_level = None # sqlite3.IMMEDIATE setattr(cls, "conn", conn) return cls def select(cls): def select(self): return cls.conn.execute(f"select * from {cls.__name__}").fetchall() setattr(cls, "select", select) return cls @select @connect class Albums:... if __name__ == "__main__": a = Albums() print(a.select()) </code></pre> 8. Even more interesting, how decorators are used elsewhere:</h1> Cubicweb</a> decorator</a> used to raise an AuthenticationError if no user is detected.</li> Flask</a> decorator</a> that redirects anonymous users to the login page.</li> Templating decorator</a> invented by the TurboGears</a> guys a while back to automatically render a template.</li> Django</a> decorators</a> to manipulate http headers</a>.</li> </ul> 9. More on this topic:</h1> I hope that you found this post helpful. To learn about this topic, please, browse the following links: PEP 318 – Decorators for Functions and Methods</a></li> PythonDecoratorLibrary</a></li> functools</a></li> decorator</a></li> DB-API 2.0 interface for SQLite databases</a></li> The difference between wraps and update_wrapper</a></li> Python Decorators III: A Decorator-Based Build System</a></li> Interior decorating</a></li> </ul> Test driven interview 2022-10-16T00:00:00+00:00 Introduction:</h1> Test Driven Development TDD is a method of implementing software that rely on software requirements being converted to test cases before software is fully developed. Red, Green and Refactor is the 3 phases of TDD. When followed, this order of steps helps ensure that you have tests for the code you are writing and you are writing only the code that you have to test for. Context:</h1> Let's suppose we should implement some requirements. Let's suppose all the requirements and specifications are written using unit tests. See this repository</a>. Our mission is to write enough code to make all the tests pass. Let's dive in</a>. TOC:</h1> Setup</a></li> Tests</a></li> Requirements</a> First requirement</a></li> Second requirement</a></li> Third requirement</a></li> Fourth requirement</a></li> Fifth requirement</a></li> Last requirement</a></li> </ul> </li> </ul> Setup</h2> >> git create-branch feat-1 # create a new branch for feature work Basculement sur la nouvelle branche 'feat-1' (.env) ~/GIT/interview (feat-1) >> python3.10 -m venv .env # create a virtual environment ~/GIT/interview (feat-1) >> source .env/bin/activate # activate virtual environment (.env) ~/GIT/interview (feat-1) >> pip install -r requirements.txt # install requirements Collecting django<3.2,>=3.1 Using cached Django-3.1.14-py3-none-any.whl (7.8 MB) Collecting djangorestframework<3.13,>=3.12 Using cached djangorestframework-3.12.4-py3-none-any.whl (957 kB) Collecting sqlparse>=0.2.2 Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB) Collecting asgiref<4,>=3.2.10 Using cached asgiref-3.5.2-py3-none-any.whl (22 kB) Collecting pytz Using cached pytz-2022.4-py2.py3-none-any.whl (500 kB) Installing collected packages: pytz, sqlparse, asgiref, django, djangorestframework Successfully installed asgiref-3.5.2 django-3.1.14 djangorestframework-3.12.4 pytz-2022.4 sqlparse-0.4.3 WARNING: You are using pip version 22.0.4; however, version 22.3 is available. You should consider upgrading via the '/home/nsukami/GIT/interview/.env/bin/python3.10 -m pip install --upgrade pip' command. (.env) ~/GIT/interview (feat-1) </code></pre> How to launch tests ?</h2> According to the documentation</a>, we can run tests using the test command of your project's manage.py utility. Let's run that command</a> when any of the Python files changes. We also tell Django to stop running the test suite after first failed test. In a different terminal, let's type one of the following commands: >> # you can ask git ls-files command to watch for all the python files >> git ls-files -- '*.py' ':!:*__.py' | entr sh -c "python src/manage.py test --failfast --force-color -v 2 src" </code></pre> >> # you can ask find command to watch for all the python files >> find src -type f -name "*.py" -not -name "__*" | entr sh -c "python src/manage.py test --failfast --force-color -v 2 src" </code></pre> >> # the ack command seems more readable >> ack -f --python | entr sh -c "python src/manage.py test --failfast --force-color -v 2 src" </code></pre> Here is our first error: django.db.utils.OperationalError: no such table: inventory_product </code></pre> 1. First requirements: we need a Product model.</h2> We need to create a model named Product inside an application named inventory. Somewhere</a> within the tests file, we can read: def create_coke(self): return Product.objects.create( description='Coca-Cola', unit_price=500, stock=10 ) </code></pre> Let's open</a> the models.py</code> file and write the following: from django.db import models class Product(models.Model): description = models.CharField(max_length=200) unit_price = models.FloatField() stock = models.IntegerField() </code></pre> After running the migration scripts: >> python src/manage.py makemigrations && python src/manage.py migrate </code></pre> The tests tells us to return an http status 201</a> after Product creation. Right now, we have a 404. We need to: create a REST API.</li> we should be able to create a Product via the API.</li> and the returned http status should be 201 _created.</li> </ul> ====================================================================== FAIL: test_create_product (modules.inventory.tests.ProductTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nsukami/GIT/interview/src/modules/inventory/tests.py", line 45, in test_create_product self.assertEqual(response.status_code, 201) AssertionError: 404 != 201 ---------------------------------------------------------------------- Ran 1 test in 0.010s FAILED (failures=1) </code></pre> We need a REST API for managing a Product.</h2> Another part</a> of the tests is telling us we need an url named api/products/</code>. Let's create a serializer</a>, a viewset</a> and a router</a>. Here is the code: # /src/modules/inventory/views.py from rest_framework import serializers # type:ignore from rest_framework import viewsets from .models import Product class ProductSerializer(serializers.ModelSerializer): class Meta: model = Product fields = "__all__" class ProductsViewSet(viewsets.ModelViewSet): queryset = Product.objects.all() serializer_class = ProductSerializer # /src/pos/urls.py from django.urls import path, include from rest_framework import routers from modules.inventory.views import ProductsViewSet router = routers.DefaultRouter() router.register(r"products", ProductsViewSet) urlpatterns = [ path("api/", include(router.urls)), ] </code></pre> After the changes, we notice some progress. The test tells us that if we attempt to delete a Product, we should receive an http status 405</a> instead of 204. Litteraly, we have been able to delete a Product while it shouldn't be allowed: ====================================================================== FAIL: test_delete_product (modules.inventory.tests.ProductTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nsukami/GIT/interview/src/modules/inventory/tests.py", line 142, in test_delete_product self.assertEqual(response.status_code, 405) AssertionError: 204 != 405 ---------------------------------------------------------------------- Ran 2 tests in 0.011s FAILED (failures=1) </code></pre> A Product cannot be deleted.</h2> I should have take the time to carefully read all the tests. Indeed</a>, one of them clearly states that we can't delete a Product. Obviously, the ModelViewSet is not a good choice. We should update our viewset in a way that forbids product deletion. The operations: creating</a>, listing</a>, updating</a>, getting one product</a> should still be permitted. To achieve that, we'll use some mixins</a> offered by the framework. The product viewset becomes: from rest_framework import viewsets, mixins class ProductsViewSet( mixins.CreateModelMixin, mixins.ListModelMixin, mixins.UpdateModelMixin, mixins.RetrieveModelMixin, viewsets.GenericViewSet, ): queryset = Product.objects.all() serializer_class = ProductSerializer </code></pre> Let's commit what have been done so far.</h3> All the tests related to Product are passing. We can commit and push what we've done so far. >> git commit -m "Add Product API" [feat-1 a11ffeb] Add Product API 5 files changed, 76 insertions(+), 6 deletions(-) create mode 100644 src/modules/inventory/migrations/0001_initial.py rewrite src/modules/inventory/views.py (98%) create mode 100644 src/modules/orders/migrations/0001_initial.py </code></pre> >> git push origin feat-1 Énumération des objets: 25, fait. Décompte des objets: 100% (24/24), fait. Compression par delta en utilisant jusqu'à 4 fils d'exécution Compression des objets: 100% (14/14), fait. Écriture des objets: 100% (14/14), 1.80 Kio | 1.80 Mio/s, fait. Total 14 (delta 6), réutilisés 0 (delta 0), réutilisés du pack 0 remote: remote: Create a new pull request for 'feat-1': remote: https://git.disroot.org/nsukami/interview/compare/master...feat-1 remote: remote: . Processing 1 references remote: Processed 1 references in total To https://git.disroot.org/nsukami/interview.git * [new branch] feat-1 -> feat-1 </code></pre> We need another model named Order.</h2> All the tests located in the inventory</a> application are passing. This is why we decided to commit and push. There is one test currently failing, it it located inside the orders</a> application. The test tells us we need to create a model named Order: ====================================================================== FAIL: test_create_order (modules.orders.tests.OrdersTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nsukami/GIT/interview/src/modules/orders/tests.py", line 87, in test_create_order self.assertEqual(response.status_code, 201) AssertionError: 404 != 201 ---------------------------------------------------------------------- Ran 7 tests in 0.054s FAILED (failures=1) </code></pre> Let's continue reading the tests. Line 90</a>, we read that an item should have 4 attributes: description, quantity, unit_price, total</code>. We also read that an Order has 2 attributes: items</code> and total</code>: order = Order.objects.get() expected = { 'id': order.id, 'items': [ { 'description': 'Coca-Cola', 'quantity': 1, 'unit_price': 500, 'total': 500, }, { 'description': 'Potato Chips', 'quantity': 2, 'unit_price': 1000, 'total': 2000, } ], 'total': 2500, } </code></pre> So, our models may look like the following: class Order(models.Model): @property def total(self): # this field is just a sum, depending on all the items in the current Order return sum([x.total for x in self.items.all()]) class OrderItem(models.Model): order = models.ForeignKey(Order, related_name="items", on_delete=models.CASCADE) # For being able to retrieve the description and the unit_price fields ? # Should I add Product as a foreign key ? # Should I duplicate the fields ? @property def total(self): # the total is obviouly a property that will be calculated # based on the number of items in the Order & the unit price for each item return float(self.unit_price * self.quantity) </code></pre> We should be able to update the stock depending on the number of items ordered:</h2> The Line 114</a> of the tests tells us, a Product should have a method to refresh its stock: self.chips.refresh_from_db() self.assertEqual(self.chips.stock, 8) </code></pre> Let's update our Product model and add the new method. This method will substract the number of Product ordered from the current stock: from django.db import models class Product(models.Model): description = models.CharField(max_length=200) unit_price = models.FloatField() stock = models.IntegerField() def refresh_from_db(self): from modules.orders.models import OrderItem self.stock = self.stock - sum( oi.quantity for oi in OrderItem.objects.filter(id=self.id) ) self.save() </code></pre> We should be able to update a Product without modifiying already made Orders:</h2> After reading the Line 141</a>, we know that the fields description and unit_price</code> will be duplicated inside the OrderItem model. During the OrderItem creation, we should copy those fields from the Product model. This task will be achieved using our serializers: class ItemSerializer(serializers.ModelSerializer): quantity = serializers.IntegerField(required=False) unit_price = serializers.FloatField(required=False) description = serializers.CharField(required=False) class Meta: model = OrderItem fields = ["description", "quantity", "unit_price", "total"] class OrderSerializer(serializers.ModelSerializer): items = ItemSerializer(many=True) def create(self, validated_data): order = Order.objects.create() # I want to be able to access the id, the id represent the Product id # So, I'm using initial_data instead of validated_data # I'll need to find a proper way for item_data in self.initial_data["items"]: product = Product.objects.get(id=item_data["id"]) OrderItem.objects.create( id=product.id, order=order, description=product.description, unit_price=product.unit_price, quantity=item_data["quantity"], ) return order class Meta: model = Order fields = ["id", "items", "total"] class OrdersViewSet( mixins.CreateModelMixin, mixins.ListModelMixin, mixins.RetrieveModelMixin, viewsets.GenericViewSet, ): queryset = Order.objects.all() serializer_class = OrderSerializer </code></pre> Conclusion:</h1> It's been a long time since I've written Django and DRF code. I really enjoyed the exercise. The final result can be found here</a>. >> python src/manage.py test src --failfast Creating test database for alias 'default'... System check identified no issues (0 silenced). ............ ---------------------------------------------------------------------- Ran 12 tests in 0.093s OK Destroying test database for alias 'default'... </code></pre> While I still have your attention:</h1> A conversation</a> with Adam Hill about writing software, the perfect settings file set up, and more. Adam is a software engineer at The Motley Fool and the author of multiple open source packages including Django Unicorn</a>. The videos for the DjangoCon Europe 2022</a> are available. Can someone tell me please when the DjangoCon Africa</a> event will be held ? So I can prepare myself ? More on the topic:</h1> I really hope you have learned something. To dive even more deeper in the topic, let me please recommend the following links: Marketing interview</a></li> Test Driven Development</a></li> Unit testing framework</a></li> Testing a Django web application</a></li> Test Driven Web Development with Django</a></li> Writing and running tests</a></li> API Improvement Proposal</a></li> Django single file project</a></li> How we write/review code in big tech companies</a></li> </ul> Linear regression 2022-07-24T00:00:00+00:00 For the Dakar Institute of Technology</a>, I give Python introductory classes. During one of my sessions I was asked about simple linear regression with Python. Let the music play</a> </center> TOC:</h1> Introduction</a></li> First solution</a> Reading the data set</a></li> Plotting the data set</a></li> Splitting the data set</a></li> Least squares estimates</a> Mean</a></li> Variance</a></li> Covariance</a></li> Estimates</a></li> Prediction</a></li> How good is the model</a></li> Regression line</a></li> </ul> </li> </ul> </li> The complete program</a></li> Another solution using the statistics module</a></li> Just give me the code please</a></li> </ul> Introduction:</h1> Simple linear regression is a commonly used type of predictive analysis. The idea of regression is to examine if a set of predictor variables do a good job in predicting an outcome variable. Linear regression models are used in many industries such as healthcare (predicting disease from biological factors), economics (predicting growth), businesses (predicting product sales), etc ... For simple linear regression, the relationship between the independent variable x and the dependent variable y is expressed as: y = slope * x + intercept + noise</code>, where: slope and intercept are the regression parameters that are estimated</li> noise is the variability of the data (difference between predicted and actual values of y).</li> </ul> First solution:</h1> Let's write a Python program to achieve this task. We'll use the following data set</a> 1. Reading the data set:</h2> To read the data, we'll write a function in charge of opening and reading the file containing our data. We assume each line will consist of an int value, followed by a tabulation and a float value. This function will: open the file</li> get all the lines except the first one (the header)</li> convert first column to int, convert second column to float</li> return a tuple containing 2 things: </li> </ul> def read_text( filename: t.Union[pathlib.Path, t.AnyStr], ) -> t.Tuple[t.Collection[float], t.Collection[float]]: with open(filename, "r") as f: slines = [line.strip().split("\t") for line in [li for li in f.readlines()[1:]]] nlines = [(int(x), float(y)) for x, y in slines] X, Y = zip(*nlines) return X, Y </code></pre> Bonjour 2. Plotting the data set:</h2> We'll have to do some plotting. The following program is just one we use to setup our plots</a>: ax: mpl.axes.Axes _, ax = plt.subplots() ax.set_xlabel("X-coordinates") ax.set_ylabel("Y-coordinates ") ax.set_title("Graphical representation of our points") ax.grid(True) </code></pre> The first step in finding a linear regression equation is to determine if there is a relationship between the two variables. To determine if there is a relationship between x and y we can scatter plot the data set. Scatter plots are used to observe relationships between variables. We'll need one function to do a scatter plot: put dots on our graph and see each individual data point. def scatter_plot( x: t.Collection[float], y: t.Collection[float], ax: mpl.axes.Axes ) -> None: ax.scatter(x, y, color="red") </code></pre> Using this function with our data set</a>, we can see the following chart: </center> We'll need a function to draw a line. If we correctly find estimates for our regression parameters (slope and the intercept), we should be able to draw a line that will go through the cloud of data points: def line_plot(a: int, b: int, ax: mpl.axes.Axes) -> None: x = [0, 130] y = [a * x + b for x in x] ax.plot(x, y) </code></pre> 3. Splitting the data set:</h2> We don’t want our model to over learn from the data and perform poorly when presented data never seen before. We also need to have a mechanism to assess how well our model is generalizing. Hence, we'll need to separate our input data into at least 2 sets: training and testing: 1. Training data set:</h4> This is the set of data that will be used only for learning. We will use only one part of the data set to find the best parameters for our model. 2. Testing data set:</h4> It would be best if we evaluate the model with new data that hasn’t been seen by the model before. So we need a set of data used to provide an evaluation without bias of the model that was fitted on the training data set. Let's write a function that will split our data. We'll keep 80% for the training and 20% for testing. This function will: randomly pick 20% of all the indexes in our collection</li> with those 20% indexes, create a list containing 20% of the collections</li> from those 20% indexes, create a list containing 80% of the collections</li> return all the sets</li> </ul> split_seq = t.Tuple[ t.Collection[float], t.Collection[float], t.Collection[float], t.Collection[float] ] def split_train_test( X: t.Collection[float], Y: t.Collection[float], frac: float = 0.2 ) -> split_seq: indexes = list(range(len(X))) indexes_test = set(random.sample(indexes, int(frac * len(X)))) # get 20% of Xs based on the indexes, do the same for Ys X_test = [n for i, n in enumerate(X) if i in indexes_test] Y_test = [n for i, n in enumerate(Y) if i in indexes_test] # get 80% of Xs based on the indexes, do the same for Ys X_train = [n for i, n in enumerate(X) if i not in indexes_test] Y_train = [n for i, n in enumerate(Y) if i not in indexes_test] return X_train, X_test, Y_train, Y_test </code></pre> Let's visualize how the 2 sets are rendered in a scatter plot: </center> 4. Least squares for the slope and the intercept:</h1> How do we estimate our parameters α and β ? That’s the learning procedure. There are different optimization techniques to implement this part. To determine the optimum values of these two parameters, we should find the line that best fit the data set. To determine the optimum values of these two parameters, we need to find the slope and the intercept so that when we sum all the errors between the predicted y values and the observed y values, this sum should be the least possible. The following chart will show us all the distances (or all the differences, or all the errors) between each point on the dataset and the regression line: </center> Let's see how the chart looks like when the parameters are not correctly estimated (the regression line does not "fit" the data set) : </center> And here is the function used to plot the errors. This function will: Take the estimated coefficients α and β</li> Take the list of X and the list of Y</li> For all points in our data set: predict y</code> based on α and β</li> draw a dashed line between 2 points: (x, y)</code> and (x, y_pred)</code></li> </ul> </li> </ul> def plot_errors( a: float, b: float, X: t.Collection[float], Y: t.Collection[float], ax: mpl.axes.Axes, ) -> None: for x, y in zip(X, Y): y_pred = a * x + b ax.plot([x, x], [y, y_pred], "bo", label="Errors", linestyle="dashed") </code></pre> The total error of this model is the sum of all the errors for each point:</h1> We should sum all the errors and look for the least error. Here is the formula: The SSE (Sum of Squared Errors) is equal to the sum of the square of Y observed - Y calculated, for all point in the data set. </center> If we do all the correct algebraic manipulation</a>, we should find that: </center> Seeing those formulas, those familiar with statistics already know we'll need to define a function to calculate a mean, another function to calculate a variance and a function to calculate a covariance. βeta hat can be defined as Covariance(X, Y) / Variance(X). 5.1 Mean:</h2> The arithmetic mean, is a measure of central tendency of a finite set of numbers: specifically, the sum of the values divided by the number of values: def mean(lst: t.Collection[float]) -> float: return sum(lst) / len(lst) </code></pre> 5.2 Variance:</h2> The variance is the measure of the spread in a data set. The variance is a measure of how far each number in the data set is from the mean, and thus from every other number in the data set. The higher the variance, the more spread out the variables are. def variance(lst: t.Collection[float]) -> float: m = mean(lst) return sum([(x - m) ** 2 for x in lst]) / len(lst) </code></pre> variance: 582.8381391772394 </code></pre> There is 2 ways to plot the variance. We can do it manually, or we can use a box plot</a>: </center> Here is the function used to plot the variance: def plot_variance( X: t.Collection[float], Y: t.Collection[float], ) -> None: fig: mpl.figure.Figure ax: mpl.axes.Axes _, ax = plt.subplots(2) ax[1].set_xlabel("X-coordinates ") ax[1].set_ylabel("Y-coordinates") ax[0].set_title("Graphical representation of variance") ax[1].grid(True) # box plot ax[0].boxplot([X], vert=False, showmeans=True) # mean line ax[1].axvline(x=mean(X), ymin=0, ymax=130, color="orange") # distance from point to mean line x_mean = mean(X) for x, y in zip(X, Y): ax[1].plot( [x, x_mean], [y, y], marker="o", color="green", label="Mean", linestyle="dotted", ) plt.show() </code></pre> 5.3 Covariance:</h2> The covariance is a measure of the relationship between two variables. Covariance is the extent to which the variance in one variable depends on another variable. The higher the covariance, the stronger the relationship. def covariance(X: t.Sequence[float], Y: t.Sequence[float]) -> float: mean_x, mean_y, length_x = mean(X), mean(Y), len(X) covar = 0.0 for i in range(len(X)): covar += (X[i] - mean_x) * (Y[i] - mean_y) return covar / length_x </code></pre> covariance: 2059.003344867359 </code></pre> 5.4 Estimates:</h2> Now that we have a function for calculating the variance and another one for calculating the covariance, we can write a function that will calculate our slope and our intercept using this 2 formulas seen earlier: def estimate_coefficients(X: t.Collection[float], Y: t.Collection[float]) -> t.Tuple[float, float]: b1 = covariance(X, Y) / variance(X) b0 = mean(Y) - b1 * mean(X) return b0, b1 </code></pre> 5.5. Prediction:</h2> Now that we have extracted our coefficients, let's see if we can make a prediction, we'll use the testing data set: b, a = estimate_coefficients(X_train, Y_train) Y_predicted = [a * x + b for x in X_test] for i, j in zip(Y_test, Y_predicted): print(i, '\t', j) # print Y observed in testing set next to Y predicted using our coefficients </code></pre> > python3 script.py 392.5 389.8167994835544 15.7 65.88576964183908 48.8 45.426967757099156 113.0 99.98377278307227 48.7 52.246568385345796 40.3 38.607367128852516 59.6 76.11517058420904 152.8 147.72097718079874 73.4 161.360178437292 21.3 59.066169013592436 55.6 48.83676807122248 194.5 123.85237498193551 </code></pre> 5.6. How good is our model:</h2> We need to able to measure how good our model is. Among the methods available to achieve this task, let's use the most common and the most used: the Root mean squared error</a>. RMSE:</h3> One way (the most common way?) to assess how well our model fits our data set is to calculate the Root Mean Squared Error. The RMSE is: the square root </li> of the mean </li> of the square of all of the errors</li> </ul> Let's write a function to calculate the RMSE: def rmse(actual: t.Sequence[float], predicted: t.Sequence[float]) -> float: sum_error = 0.0 for i in range(len(actual)): prediction_error = predicted[i] - actual[i] sum_error += prediction_error**2 mean_error = sum_error / float(len(actual)) return math.sqrt(mean_error) </code></pre> If we develop mutliple models, and we want to verify which one performs better: Calculate the RMSE value for each model</li> The lower the RMSE value, the better the model. </li> A RMSE value of 0 indicate a perfect fit. </li> </ul> The data set we're working with has a range: [Y_min, Y_max]. This range is important in determining whether a given RMSE value is "low" or "high". There is no "good" or no "bad" RMSE value. It always depends on the range of the data set we're working with. Normalized RMSE:</h3> The NRMSE is the RMSE divided by Y_max - Y_min. The result will be between 0 and 1, where values closer to 0 represent better fitting models. The normalization facilitates the comparison between data sets with different scales: def rmse( actual: t.Sequence[float], predicted: t.Sequence[float] ) -> t.Tuple[float, float]: mean_error = sum([(predicted[i] - actual[i])**2 for i in range(len(actual))]) / float(len(actual)) rmse = math.sqrt(mean_error) nrmse = rmse / (max(predicted) - min(predicted)) return rmse, nrmse </code></pre> >> python3 script.py The RMSE is 23.27666203730159 Normalized RMSE is 0.06371542339113609 </code></pre> 5.7. Regression line:</h2> </center> The complete program:</h1> import math import random import requests # type:ignore import matplotlib as mpl # type:ignore import matplotlib.pyplot as plt # type:ignore import typing as t import pathlib from dataclasses import dataclass, field # setup plots fig: mpl.figure.Figure ax: mpl.axes.Axes fig, ax = plt.subplots() ax.set_xlabel("X-coordinates ") ax.set_ylabel("Y-coordinates") ax.set_title("Graphical representation of our points") ax.grid(True) splits = t.Tuple[ t.Collection[float], t.Collection[float], t.Collection[float], t.Collection[float] ] @dataclass class SimpleLinearRegression: filename: t.Union[pathlib.Path, str] = "./dataset.txt" __X: t.Collection[float] = field(init=False) __Y: t.Collection[float] = field(init=False) X_train: t.Collection[float] = field(init=False) Y_train: t.Collection[float] = field(init=False) X_test: t.Collection[float] = field(init=False) Y_test: t.Collection[float] = field(init=False) def __post_init__(self) -> None: with open(self.filename, "r") as f: slines = [ line.strip().split("\t") for line in [li for li in f.readlines()[1:]] ] nlines = [(float(x), float(y)) for x, y in slines] self.__X, self.__Y = zip(*nlines) self.__split_train_test() def scatter_plot(self, ax: mpl.axes.Axes, color: str = "red", label=None) -> None: ax.scatter(self.__X, self.__Y, color=color, label=label) def line_plot(self, ax: mpl.axes.Axes) -> None: b, a = self.estimate_coefficients() x = [0, 130] y = [a * x + b for x in x] ax.plot(x, y, color="green") def __mean(self, lst: t.Collection[float]) -> float: return sum(lst) / len(lst) def __var(self, lst: t.Collection[float]) -> float: m = self.__mean(lst) return sum([(x - m) ** 2 for x in lst]) / len(lst) def __cov(self, X: t.Sequence[float], Y: t.Sequence[float]) -> float: mean_x = self.__mean(X) mean_y = self.__mean(Y) covar = 0.0 for i in range(len(X)): covar += (X[i] - mean_x) * (Y[i] - mean_y) return covar / len(X) def __split_train_test(self, frac: float = 0.2) -> None: indexes = list(range(len(self.__X))) indexes_test = set(random.sample(indexes, int(frac * len(self.__X)))) # get 20% of Xs based on the indexes, do the same for Ys self.X_test = [n for i, n in enumerate(self.__X) if i in indexes_test] self.Y_test = [n for i, n in enumerate(self.__Y) if i in indexes_test] # get 80% of Xs based on the indexes, do the same for Ys self.X_train = [n for i, n in enumerate(self.__X) if i not in indexes_test] self.Y_train = [n for i, n in enumerate(self.__Y) if i not in indexes_test] def estimate_coefficients( self, ) -> t.Tuple[float, float]: mean_x, mean_y = self.__mean(self.X_train), self.__mean(self.Y_train) b1 = self.__cov(self.X_train, self.Y_train) / self.__var( # type:ignore self.X_train ) b0 = mean_y - b1 * mean_x return b0, b1 def rmse(self) -> t.Tuple[float, float]: b, a = self.estimate_coefficients() predicted = [a * x + b for x in self.X_test] actual = self.Y_test mean_error = sum( [(predicted[i] - actual[i]) ** 2 for i in range(len(actual))] # type:ignore ) / float( len(actual) ) rmse = math.sqrt(mean_error) nrmse = math.sqrt(mean_error) / (max(predicted) - min(predicted)) return rmse, nrmse # program entry point if __name__ == "__main__": def main() -> None: model = SimpleLinearRegression() model.scatter_plot(ax) model.line_plot(ax) b, a = model.estimate_coefficients() rmse, nrmse = model.rmse() print(f"The slope is {a}, the intercept is {b}") print(f"The RMSE and NRMSE are: {rmse} and {nrmse}") plt.show() main() </code></pre> The slope is 3.426108589853597, the intercept is 20.478117458572427 The RMSE and NRMSE are: 27.82841276749733 and 0.07520791645397 </code></pre> Another solution, using the statistics module:</h1> Before jumping to big and powerful libraries such as NumPy</a>, SciPy</a>, Pandas</a>, Pingouin</a>, Statsmodel</a>, and all the others. We have to notice, Python provides us with a module named statistics</a>. Even though the statistics module is not intended to be a competitor to the ones listed above, it's interesting to see that inside, there is: a mean</a> function</li> a variance</a> function</li> a covariance</a> function</li> even a linear regression</a> function.</li> </ul> Our program will suddenly become shorter: import statistics as lestat # load dataset X, Y = read_text("dataset.txt") # extract coefficients a, b = lestat.linear_regression(X, Y) </code></pre> Source code:</h1> Phew</a>. All the cool devs today will share the source code via a Jupyter notebook stored on Github or on Colab. According to the Software Freedom Conservancy</a>, we should all Give Up GitHub</a>. And as a free software advocate, I agree with them. The source code can be found here</a> and the data set here</a>. You're welcome. More on this topic:</h1> I really hope you've learned something, really. Talking about the whole subject won't be possible in this tiny article. To dive deeper and learn even more, let me please recommend the following links: Datasette</a></li> Matplotlib</a></li> Real programmers</a></li> Root-Mean-Squared Error</a></li> RStudio is becoming Posit</a></li> Ordinary least squares Linear Regression with scikit learn</a></li> </ul> Scafold 2022-07-24T00:00:00+00:00 It's been a very long time since I've written my first post</a> about the way I use make. Now that I'm reading it again, I realize I don't even use Makefiles anymore. Since few years now, I use Tox</a>, too much Python development I guess. I started (again) learning C programming, I had to write a simple Makefile for being able to easily and quickly build, install, remove my executables. The following is a Bash script I've created to bootstrap small C projects: #!/bin/bash GIT=`which git` MSG="initial commit from $USER on $(date)" CURRENT_DIRECTORY=`pwd` PROJECT_NAME=$1 EXECUTABLE_NAME="${2:-kli}" if [[ -z "${PROJECT_NAME}" ]]; then echo "Please add the name of the project" exit fi if [[ -d "${CURRENT_DIRECTORY}/${PROJECT_NAME}" ]]; then echo "The folder exists. Choose another name" exit fi # project tree mkdir -p "${CURRENT_DIRECTORY}/${PROJECT_NAME}"/{build,src,docs,include,lib,tests,data} # clang format config ??? clang-format --style=llvm -dump-config > "${CURRENT_DIRECTORY}/${PROJECT_NAME}"/.clang-format # Makefile header cat <<EOF >> "${CURRENT_DIRECTORY}/${PROJECT_NAME}"/Makefile SHELL := /bin/sh EXECUTABLE := $EXECUTABLE_NAME EOF # Makefile body cat <<"EOF" >> "${CURRENT_DIRECTORY}/${PROJECT_NAME}"/Makefile # https://matt.sh/howto-c # where all bin will be moved for easy trying BIN_DIR := $(shell echo $${HOME})/bin # set build and source directories BUILD_DIR := build SRC_DIRS := src LIBS := -lm -lpthread # all source files are .c files under source directory SRCS := $(shell find $(SRC_DIRS) -name *.c) # all objects files are obtained from all the .c files in src directory # and placed under build directory with the o extension OBJS := $(SRCS:%.c=$(BUILD_DIR)/%.o) # deps are .d files, obtained from all the .o files in the build directory DEPS := $(OBJS:.o=.d) # directories to include are all the directories under src directory INC_DIRS := $(shell find $(SRC_DIRS) -type d) # add -I in front of each subdirectory under src directory INC_FLAGS := $(addprefix -I,$(INC_DIRS)) # CFLAGS ?= $(INC_FLAGS) -g -Wall -Wfatal-errors -Wpedantic -Wextra -Werror -Wshadow -std=c11 -MMD -MP -march=native install: $(BUILD_DIR)/$(EXECUTABLE) ## install prog @echo "installing..." $(CP) $(BUILD_DIR)/$(EXECUTABLE) $(BIN_DIR) $(BUILD_DIR)/$(EXECUTABLE): $(OBJS) ## build executable @echo "building executable file..." $(CC) $(OBJS) -o $@ $(LDFLAGS) $(LIBS) # CC and LDFLAGS are predefined see make -p -f /dev/null $(BUILD_DIR)/%.o: %.c clean ## build object files @echo "building object files..." @$(MKDIR_P) $(dir $@) # $< evaluates to the first "prerequisite", $@ evaluates to the "target" $(CC) $(CFLAGS) -c $< -o $@ format: @echo "reformatting..." find . -iname *.h -o -iname *.c | xargs clang-format -style=file -i -fallback-style=none clean: ## delete build files @echo "deleting old build..." @$(RM) -r $(BUILD_DIR) @$(RM) -r $(BIN_DIR)/$(EXECUTABLE) # acts like include except that there is no error (not even a warning) if DEPS does not exists, sinclude is another name -include $(DEPS) # Meh :\ MKDIR_P ?= mkdir -p CP ?= cp -r .PHONY: clean install help format EOF # main source file cat <<EOF > "${CURRENT_DIRECTORY}/${PROJECT_NAME}"/src/main.c #include <stdio.h> #include <stdlib.h> #include <getopt.h> #include <stdbool.h> #include <string.h> #include <stdint.h> int main (int argc, char **argv) { printf("Hello World.\n"); exit(EXIT_SUCCESS); } EOF # git it cd ${PROJECT_NAME} ${GIT} init -q . ${GIT} add --all . ${GIT} commit -m "$MSG" </code></pre> </center> Yasnippet 2022-07-24T00:00:00+00:00 The following is a Yasnippet I've created to quickly insert getzola</a> front matters. # -*- mode: snippet -*- # name: Zola header # key: zheader # -- +++ title = "$1" date = ${2:`(format-time-string "%Y-%m-%d")`} updated = ${3:`(format-time-string "%Y-%m-%d")`} slug= $4 draft = true [taxonomies] categories= [${5:"Python"}] tags= ["$6"] +++ $0 # More on this topic: I really hope you've learned something reading this article. Talking about the current subject won't be possible in one tiny article. If you want to dive deeper and to learn even more, let me recommend the following links: - [Foo](#) - [Bar](#) - [Baz](#) </code></pre> Wall of Fame 2022-07-04T00:00:00+00:00 Here it is, the Wall of Fame. </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </a> </div> Writing 2022-05-20T00:00:00+00:00 I often receive resumes from people looking for a job and asking me for help. To all of them I always ask: "Do you have a website ?", "Do you write ?". And most of the time, the answer is negative. Writing skills are important because they allow the reader to get their first impression of who is the writer. For example: writing skills displayed in resume, cover letter email communications, website. In my humble opinion, if you're looking for a job as a software engineer, you should have a tiny space on the internet. A space in which you write about what you experiment, what you code, what you build. This article is an attempt to answer, why I think everybody (every software engineer) should have a website. Or at least, why everybody should write. This article is also kind of a recap of a talk I gave for the Dakar Institute of Tecnology</a>. </center> Some reasons for you to start writing: </h1> 1. Improve your knowledge on a topic</h2> You'll often have to write about subjects that you are not familiar with, so finding quality sources quickly is a skill you'll develop. Before you write a single word, you need to do your research about the topic you're writing on. The more you do research, the more you find quality sources, the more you improve your knowledge. 2. Organize your thoughts</h2> You can write down more than you can easily remember, so that your capacity to consider a number of ideas at the same time is broadened. Once those ideas are written down, you can move them around, change them. You can also reject those that are finally not good enough. You will stay with good, original clearly defined ideas. And you will be able to organize and communicate them easily. 3. Share your expertise, meet fellow specialist and grow your knowledge</h2> Knowledge is the only thing that grows when shared. Sharing your knowledge helps you grow your knowledge by doing research, synthesizing multiple and different viewpoints. Sharing your expertise means inviting people to read you, inviting people to a new conversation. A conversation is always an occasion to meet new people and to grow as a human. 4. Make money</h2> Even professions that don't focus on writing require written communication skills. The tech industry for example, is more and more looking for people who are able to write. Those writings include things like instruction manuals, user manuals, journal articles, product descriptions, memos, wikis, documentations, etc... You can have a look : here</a></li> then here</a></li> here again</a></li> also here</a></li> </ul> By the way: Real Python is hiring</a>.</li> LWN is hiring too</a></li> </ul> disclaimer: I'm not paid by anybody to share those job offers What are the topic to write about: </h1> Coming up with writing inspiration is tough. My advice would be, in all the cases, always write about something you really like, or something you really enjoy. As a technical writer, you can write different kinds of articles: 1. Story</h2> Human beings are interested in stories about people. If your reader knows you a little through your blog, the reader will remember who you are and the reader will be more interested by your technical articles. Here</a> you'll find different kinds of stories. 2. Opinion</h2> Giving an opinion is a common way of interacting with other people. Of course you don't simply share an opinion, you'll also share the compelling and interesting argument that support this opinion. Also, please don't spend your time attacking others people or spreading controversial ideas. Here is an axample</a> of a shared opinion. 3. HOW-TOs</h2> An how-to is a document providing detailed and practical step-by-step guide on how to do something. In a precise, step-by-step approach, you should walk your reader through the process. You should make sure your reader can reproduce the same results by following the provided steps. Example</a> of an HOWTO. 4. How something work</h2> There are people like me who want to know how almost everything works. This kind of article is suitable when you want to deep dive into a very specific topic. For example, somebody already using Django sessions, may be suddenly wondering how does Django sessions work under the hood ? Here is an article</a> showing how Django sessions work. 5. Reviews</h2> You have read, used, tested, played with something. In a review, your attempt will be to write about what was your experience. You'll describe the thing, you'll share what were your feelings about the thing, you'll show the strengths and the weaknesses of the thing. Example of review</a>. Identifying the target audience: </h1> Identifying your intended audience will help you write effective content that achieves your goals. What are you writing ? Who are you writing for ? Knowing your audience will help you adapt your content accordingly. And if you don't know who to write for, then the following is a serie of questions you can ask yourself: Who am I writing for ?</li> What subject am I writing about ?</li> What does my audience already know about the subject?</li> What does my audience want to know about the subject?</li> What are the questions will my audience have?</li> </ul> Finding the audience: </h1> For example, as a Python specialist, the audience can be found in different places. Pycoder weekly news want to hear from you about projects you are working on, conferences you are running, and articles you want to share</a>. Awesome Python is another newsletter who want you to contribute</a> links to blog posts, tutorials, libraries, events, videos, books. You can als share your posts on Lobsters</a>, on Tilde News</a>, on dev.to</a>, on towardsdatascience</a>. Improving your writing: </h1> The following is what I recommend. In all the cases, you should find what is best for you. 1. Practice</h2> The most effective writers write every day, at least a bit. The more you write, the more your brain form connections between neurons. Those connections get stronger the more often you write. After some time and regular practice, you no longer have to think about it. This is when writing becomes a habit. 2. Make time</h2> Prepare to spend between 90 minutes and 3 hours writing. But hey, 15 minutes can be useful if you do it every day. Don't fool yourself, thinking you'll wait for a big chunk of free time to start. That will never happen. 3. When</h2> You are much smarter and more resilient after you have slept properly and ate. Again, find what suits you best. According to some studies, creativity peaks in the morning. My experience tells me that's true. 4. Frequency</h2> Write only when you feel it. Tools for writing... SSG:</h1> You should provide yourself with the right technology. Here again, I can tell you what I do, but there are plenty of differents tools, you should</a> always</a> find</a> what</a> suits</a> you</a> best</a>. I've choosen to build my website using a Static Site Generator (SSG). An ssg is a tool that generates a full static HTML website based on raw data (text files) and a set of templates (html, css, js files). : </center> Image source</a> Why SSG:</h1> No need for internet for writing, no need to login somewhere before writing.</li> no need for a particular environment, just a text editor, and a browser.</li> No vendor lock in, no dependency on any product for writing or migrating your content.</li> No need for a program to dynamically generate webpages. Web pages load faster.</li> No dependencies running on the server, just a web server. Increased security</li> </ul> I'm convinced, can you show me an example please:</h1> The article</a> you're reading right now, was generated from a markdown file</a> using an SSG named Zola</a> and a theme named Oceanic Zen</a>. To finish this article, I would like to quote Leslie Lamport</a> "If you’re thinking without writing, you only think you’re thinking." </blockquote> — Leslie Lamport </blockquote> Links:</h1> To learn more on this topic, I strongly recommend the following links: Why engineers should focus on writing</a></li> Think and write, with Leslie Lamport</a></li> Jordan Peterson on the importance of reading and writing</a></li> How writing online made me a millionaire</a></li> How to improve your clarity of thought ("writing is thinking")</a></li> Programmers that don't blog should start right now</a></li> Benefits of writing</a></li> Technical blogging for Python programmers</a></li> Look deeper -- write -- the wonders of writing</a></li> What nobody tells you about documentation</a></li> Spoken words fly away, written words remain</a></li> Circadian rythm</a></li> </ul> Links again:</h1> Static Site Generators</a></li> How to choose the right static site generator</a></li> Zola, your one-stop static site engine</a></li> Publii, Static Site Generator with GUI</a></li> Hugo, the world’s fastest framework for building websites</a></li> Logarion, a suite of tools, for discovering, collecting & exchanging texts</a></li> </ul> Even more links:</h1> Things that were discussed during the talk: Literate programming</a></li> Knuth on Literate Programming</a></li> Babel: active code in Org-mode</a></li> Pandoc, a universal document converter</a></li> Quarto, open-source scientific and technical publishing system built on Pandoc</a></li> </ul> </center> Function pointers 2022-04-29T16:15:50+00:00 In Python, we say, functions are first class citizen. This means, functions can be used like any other objects. They can be stored inside variables, they can be passed as argument to a function, they can be returned from a function. The following is a very basic example: square = lambda x: x * x cube = lambda x: x * x * x operations = {1: square, 2: cube} choice = int(input("Square (1) or Cube (2) ?: ")) value = int(input("Value ?: ")) print(operations[choice](value)) </code></pre> Let's start with a very simple function to print out a simple "hello world" using a function pointer: #include<stdio.h> // defining a function ptr void (*greet_ptr)(); // a function taking no argument and returning nothing void greet(){ printf("Hello world!\n\n"); } int main(){ // make our function pointer points to greet function greet_ptr = &greet; // call greet function via the function pointer greet_ptr(); return 0; } </code></pre> Slowly please?</h3> The first interesting line is: void (*greet_ptr)(); </code></pre> We're defining a pointer to a function, named greet_ptr.</li> We're using the * notation to signify that it is a pointer.</li> The function pointed to, should receive no argument: () </li> The function should return nothing: void.</li> We need parenthesis, otherwise it becomes void *greet_ptr(), a function returning a void pointer.</li> </ol> The next interesting lines are: greet_ptr = &greet; greet_ptr(); </code></pre> greet_ptr</code> is a pointer, so it should receive the address of a function.</li> greet</code> is a function, taking no argument, and returning void.</li> greet_ptr()</code>: let's execute the code living at the address stored by greet_ptr.</li> </ol> What if the function pointed to, takes arguments?</h3> Simple example: #include<stdio.h> // defining a function ptr void (*greet_ptr)(char *); void greet(char *name){ printf("Hello %s!\n\n", name); } int main(){ // make our function pointer points to greet function greet_ptr = &greet; // call greet function via the function pointer greet_ptr("foo"); return 0; } </code></pre> The first interesting line is: void (*greet_ptr)(char *); </code></pre> We're defining a pointer to a function, named greet_ptr</li> We're using the * notation to signify that it is a pointer.</li> The function pointed to, should take one argument of type char pointer (char *).</li> The function pointed to, should return nothing: void.</li> We need parenthesis, otherwise it becomes void *greet_ptr(char *), a function taking a string and returning a void pointer.</li> </ol> The next interesting lines are: greet_ptr = &greet; greet_ptr("foo"); </code></pre> greet_ptr still points to greet function.</li> let's execute the code living at the address stored by greet_ptr.</li> This executable, greet</code> is waiting for a parameter of type char *</code>.</li> So the parameter "foo"</code> will be passed to the function greet</code> before execution.</li> </ol> What is the address of a function?</h3> Before going further, we have to keep one thing in mind. a function name (label) is the address of the function. Yes. A function name (label)</code> is converted to a pointer to itself</code>. This means that function names can be assigned to a function pointer without using &</code> to pass an address. Example: #include<stdio.h> void (*greet_ptr)(char *); void (*hello_ptr)(char *); void greet(char *name){ printf("Hello %s!\n\n", name); } int main(){ greet_ptr = &greet; hello_ptr = greet; // notice, there is no & if(greet_ptr == hello_ptr) printf("Pointers containing the same address.\n"); greet_ptr("foo"); hello_ptr("foo"); return 0; } </code></pre> Function pointers as arguments?</h3> Remember, a function name (label) is converted to a pointer to itself, so it can be passed whenever we need a function pointer. So, we can write function taking function pointers as arguments. Example: #include<stdio.h> void sayHello(char *name){ printf("Hello %s!\n\n", name); } void sayHi(char *name){ printf("Hey %s!\n\n", name); } // a function that takes a function and a string as parameters void greet(void (*fctPtr)(char *), char *name) { fctPtr(name); } void greetBar(){ greet(sayHello, "bar"); } int main () { greet(sayHello, "foo"); greet(sayHi, "baz"); greetBar(); return 0; } </code></pre> The first interesting lines are: void greet(void (*fctPtr)(char *), char *name) { fctPtr(name); } </code></pre> We're defining a function, named greet</code>.</li> The greet</code> function takes 2 arguments: a function pointer: void (*fctPtr)(char *)</code></li> a char pointer: char *name</code></li> </ol> </li> The function pointed to, should be called with an argument of type char *</code></li> The function pointed to is then called: fctPtr(name)</code></li> </ol> The next interesting line is: greet(sayHello, "foo"); </code></pre> sayHello</code> is a function name (label) that will be converted to a pointer to itself.</li> then, that pointer will be passed to greet</code> function.</li> </ol> </li> The function sayHello</code> will be called with "foo"</code> as parameter.</li> </ol> We can also use function pointers as return values.</h3> We already know a pointer can be returned from a function. A function pointer, is a pointer, so there is no big deal here. The original Python example in C?</h3> #include<stdio.h> int square (int x) { return x * x; } int cube (int x) { return x * x * x; } int main () { int choice, value; // an array of pointers to functions // those functions should take an int and return an int int (*fp[2])(int) = {square, cube}; printf("Square (1) or Cube (2) ?: "); scanf("%d", &choice); printf("Value ?: "); scanf("%d", &value); printf("%d\n", (*fp[choice-1])(value)); return 0; } </code></pre> Any real life examples?</h3> Function pointers gives C programmers, first-class citizen functionality</a>: being able to pass functions as argument to other functions, and being able to return functions as value from other functions. More on the topic:</h3> C syntax for functions returning function pointers</a></li> </ul> My journey to type checking 7521 lines of Python 2021-11-08T03:40:17+00:00 I've spent the last couple of months adding type hints</a> to the RQL package</a>. In the first</a> part, I talked you the reasons I think Mypy may be useful. In this part, I'll talk about the process I've followed for adding type annotations to RQL. Recap of the first part:</h2> Type hints are mainly useful for the maintainers</a>, the human readers of large code bases. The people who read and debug and maintain large software systems they did not write. </blockquote> The major benefit for you won't be fewer bugs or unit tests, but the ability to more easily read and reason about code written by someone else. </blockquote> Outline:</h2> First, I'll talk about what Mypy</a> is and what should be it's initial setup</a>. Secondly, I'll briefly show you how I deconstructed the project (import structure</a> & classes hierarchy</a>) to have another perspective of how the project is built. I will also show 3 different ways to generate</a> type annotations and where</a> I think those type annotations should reside. Finaly, I'll talk about things that still need to be improved</a>. Mainly, the use of Any</code> and type:ignore</code>. Feel free to jump straight to the part that's most interesting. Mypy:</h2> Mypy</a> is a library that provides optional static type checking. Unlike other programming languages, where the static type-checking takes place at compilation time, Mypy CLI does the type-check on-demand. At Logilab, the type-check was done, locally when developing, but also in the Continuous Integration pipeline, using Gitlab and Tox. Initial setup:</h2> We need to ensure that Mypy is really being run everywhere it should be and everytime someone pushes code. We don't want to spend time applying type annotations that won't be checked. If you don't check type annotations on a regular basis, they may become wrong after a certain period of time. That means: 1. Configuring Mypy</h3> Configuration is done using a mypy.ini</code> file</a>. At the begining, we do not want strict typing checks. Then, we'll try to work towards more strigency, for example: dissallowing dynamic typing</a> or disallowing untyped definitions</a>. We're working with a legacy code base with no type definitions, let's start with more relaxed options. 2. Ignoring external libraries:</h3> I've done it many weeks later, but I could have done it right from the beginning: tell Mypy to ignore</a> missing type hints from external libraries</a>. Maybe in another phase, we'll annotate them, but for now, they should be ignored. 3. Set a Tox environment and CI job:</h3> Add Tox environment to automate the boring stuff and reduce the boilerplate</a>. And add a Gitlab CI job</a>. that will be in charge of running Mypy every time a new changeset is sent to the Gitlab server. Import structure</h2> I thought a diagram showing the import structure would help me understand the project. I also thought making a graph of the RQL's import structure would help me add types a bit faster. My idea was to add annotations from bottom to top. I was wrong. More than once, after adding type annotations to a module, I was obliged to go back to where it was already done. For example, method overrides</a>. PyDeps</a> is the module dependency visualization I used, the result being: If you open the diagram in another tab, you'll notice that we can see external libraries too, for example: logilab-database</a>, yapps</a>, pygments</a>. Those are the ones that need to be ignored during type checking. Classes hierachy:</h2> Within Pylint</a>, there is a tool named Pyreverse</a> that analyses Python code and extracts UML class diagrams and package dependencies. Using Pyreverse, we can have a clear view of all the classes and the relation between them: Type hints addition:</h2> There are different options for adding type annotations. 1. Manual</h3> We can do it manually: finding what are the types flowing through our program, I've been there, it's painful. Literally: Adding breakpoints here and there</li> Running the function or running the tests</li> Once inside PDB, asking the type</code> function to tell me who is who</li> </ul> In [73]: def g(x): ...: breakpoint() ...: x[5] = 'foo' ...: y = 42 ...: # a long list of non understandable code ...: return y ...: In [76]: # let's assume we launched the tests and somewhere, g is called In [77]: g([16, 72, 38, 45, 21, 34]) > <ipython-input-75-ef8444a3ab59>(3)g() -> x[5] = 'foo' (Pdb) type(x) # please, what is the type of the parameter x? <class 'list'> (Pdb) n > <ipython-input-75-ef8444a3ab59>(4)g() -> y = 42 (Pdb) n > <ipython-input-75-ef8444a3ab59>(6)g() -> return y (Pdb) type(y) # please, what is the type of the returned value? <class 'int'> (Pdb) </code></pre> After that first run, we may be temped to write our function like one that takes a list and returns an integer: In [80]: # g takes a list and return an integer In [81]: def g(x: list) -> int: ...: x[5] = 'foo' ...: y = 42 ...: # a long list of non understandable code ...: return y ...: </code></pre> I said "temped" because, maybe the function takes a dictionary and returns an integer. Only an extensive set of tests will tell us what's the best answer. 2. Reveal:</h3> I never really use them, but another option is to use reveal_type</code> or reveal_locals</code> functions</a>. You just put a reveal_type(expr)</code> in the code, and run it with Mypy. I never really use those 2 functions because most of the time, the revealed type is Any</code>: 3. PyAnnotate:</h3> The third and really helpful option I want to suggest is, using a tool that will give you an exhaustive list of suggestions, for example PyAnnotate. PyAnnotate</a> will help you list all the call arguments and all the return types observed at runtime. Similar projects exists, MonkeyType</a> created by Instagram. Pytype</a> from Google. And Pyre</a> created at Facebook... Hem, I mean, Meta. I've used PyAnnotate just because it felt like the most easiest to setup and to use. I should probably give a try to the other options. In the repository, there is a configuration example</a> you can use with Pytest. And after running your tests, you should find a new file named type_info.json</code>. This file will contain a json that looks like: { "path": "rql/__init__.py", "line": 128, "func_name": "RQLHelper.compute_solutions", "type_comments": [ "(rql.stmts.Union, None, None, int) -> Set", "(rql.stmts.Insert, None, None, int) -> Set", "(rql.stmts.Union, Dict[str, function], None, int) -> Set", "(rql.stmts.Delete, None, None, int) -> Set", "(rql.stmts.Set, None, None, int) -> Set", "(rql.stmts.Union, Dict[str, function], Dict[str, str], int) -> Set[str]", "(rql.stmts.Union, None, None, int) -> pyannotate_runtime.collect_types.NoReturnType", "(rql.stmts.Insert, None, None, int) -> pyannotate_runtime.collect_types.NoReturnType" ], "samples": 65 }, { "path": "rql/__init__.py", "line": 152, "func_name": "RQLHelper.simplify", "type_comments": [ "(rql.stmts.Union) -> None" ], "samples": 14 }, </code></pre> Those generated type-hints are suggestions. You will have to tweak them first, then apply them. Once applied, you should run mypy on the updated files to verify that everything stays green. How much should be typed?</h2> I won't deny, I was gradually</a> adding annotation to everything, everywhere. There was a feeling of endlessness. Then I asked myself: "How do I know the RQL package has enough annotations? How do I know that I've done enough?" I decided to follow</a> a path: adding Mypy coverage reports</a> to the project. The lower the percentage, the happier you should be: Mypy Type Check Coverage Summary ================================ Script: index +---------------------+-------------------+----------+ | Module | Imprecision | Lines | +---------------------+-------------------+----------+ | rql | 29.63% imprecise | 324 LOC | | rql._exceptions | 0.00% imprecise | 48 LOC | | rql.analyze | 62.35% imprecise | 603 LOC | | rql.base | 13.19% imprecise | 288 LOC | | rql.interfaces | 11.54% imprecise | 78 LOC | | rql.nodes | 24.20% imprecise | 1438 LOC | | rql.parser | 77.19% imprecise | 1245 LOC | | rql.parser.__main__ | 19.30% imprecise | 57 LOC | | rql.pygments_ext | 23.21% imprecise | 56 LOC | | rql.rqlgen | 1.62% imprecise | 247 LOC | | rql.rqltypes | 12.92% imprecise | 178 LOC | | rql.stcheck | 49.25% imprecise | 867 LOC | | rql.stmts | 30.98% imprecise | 1372 LOC | | rql.undo | 27.39% imprecise | 387 LOC | | rql.utils | 20.72% imprecise | 333 LOC | +---------------------+-------------------+----------+ | Total | 38.64% imprecise | 7521 LOC | +---------------------+-------------------+----------+ </code></pre> Mypy does type inference</a>, it can guess the types of values based on the context. We see in the picture below that even though there is no type annotations, the program is not flagged as 100% imprecise: I finally gave up</a>, to focus on what other Logilabians, suggested: find</a> and list</a> what are the RQL's most imported/used classes and functions. If those classes and functions are typed, we'll consider we've reach a milestone. Type information inline or via stubs?</h2> Mypy gives you the ability to add type annotations to your project without ever modifying the original source code. You do it using what's called "stub files". Stub</a> files are python-like files, that only contain type-checked variable, function, and class definitions. Nothing else. I went the other way, I've added type annotations directly within the original source code. In the beginning, it really felt like the easiest thing to do, having everything in the same file. But now that there is more and more types, I think I'm better improve readability. I can for example write type aliases</a> whenever necessary</a>. Interesting inspiration can be found here</a>. I can also write a snippet that will strip all those annotations and put them inside .pyi</code> files. From now on, I think I'll always use stub files. Type Checking:</h2> The typing module defines a TYPE_CHECKING</a> constant that is False</code> at runtime but treated as True</code> while type checking. This technique is used to tell mypy something without the code being evaluated at runtime, for example resolving import cycles. Example here</a>, another one here</a>. Improvements:</h2> 1. Less Any:</h3> I've been into situations where I'm not sure what should be the correct type annotation. Instead of leaving an empty type annotation, I put Any</code>, to show other developers that the lack of a restrictive type hint is a conscious choice: either I don't know, either I'm not sure. Normally, I should use Any</a> only if it's impossible for me to know what the type is going to be. From the documentation we can read</a>: Notice that no type checking is performed when assigning a value of type Any to a more precise type. For example, the static type checker did not report an error when assigning a to s even though s was declared to be of type str and receives an int value at runtime! </blockquote> So, to take advantage of Mypy, I should really avoid using Any</a> and see if I can replace it with a proper annotation. 2. Less type: ignore comments:</h3> It was also impossible to progress without using # type: ignore</code> comments, to work around the trickier cases. There are still too many type: ignore</a> comments. I should have another look and see if I can improve the type annotations and remove some of those type: ignore comments. Thanks:</h2> Special thanks to Romaric</a> who took the time to read this article and provide insightful critiques. More on this topic:</h2> Really hope you've learned something reading this article. As always, the topic is wide and cannot be completely covered here. To discover more about this subject, I strongly recommend the following links: Awesome typing</a></li> Liskov Substitution Principle with Johan Vergeer</a></li> Our journey to type checking 4 million lines of Python</a></li> Static analysis at scale: an Instagram story</a></li> </ul> To type or not to type? That is the question 2021-08-22T03:40:17+00:00 I've spent the last couple of months adding type hints</a> to the RQL package</a>. In this first part, I'll try to tell you when I think Mypy may be useful. In a second part</a>, I'll talk about the how, the process I've followed for adding type annotations to RQL. In a last part, I'll talk about what I've learned during the process. What?</h2> Yes, in case you were living under a rock, here are the news: you can now write Python as if it was a static programming language. The official PEP</a>, the official website</a>. The official documentation</a>. And the official repository</a> 1</a>. The Alore</a> programming language, the project Mypy borrows heavily from. Why?</h2> I remember when I was asked to work on this task. The (my) skepticism was real. Around me, reactions were mixed (confused, disgusted, interesed). I won't deny, sometimes, I'm really wondering if I'm not better go and learn to write OCaml</a>. Python devs are so used to quickly prototype their programs, without worrying about the types of their variables, without worrying about their return types, etc. You cannot just come and tell them that type hints are nice! They need to be convinced. More than once, the discussion ended with the "explicit vs implicit", "static vs dynamic" holy wars, etc... I'm not trying to fuel those debates. There are pros and cons on each sides. You have to know what are the tools, what suits you best, depending on what you're trying to achieve, depending on what is your position (user, author, maintainer). Assuming you're a person who already write Python programs, and you want to discover more about Mypy. If you only look at some introductory examples, they will help you understand what's going on. They won't convince you to jump on the type-hints band wagon. The problem with most of the examples is, they're too short, or too basic. No disrepect to the authors, I guess, the intent was to make things easier to understand. Short and simple to get to the point. But short and simple would barely convince you to try. So, what are the situations in which adding types to your program will be useful, necessary, not a waste of time, priceless, meaningful? To this question, I would say this: Type hints are mainly useful for the maintainers</a>, the human readers of large code bases. The people who read and debug and maintain large software systems they did not write. </blockquote> The major benefit for you won't be fewer bugs or unit tests, but the ability to more easily read and reason about code written by someone else. </blockquote> I need to know the types:</h2> When you read/debug/maintain a large software system you've not written, you'll often be in the position in which you ask yourself: "Those function arguments, those variables, what are their types?", or "Is this function returning something?". Remember, you were reading the source code, now you need to: modifiy the source code, add a couple of: print(foo)</code>, type(bar)</code>, isinstance(baz)</code>, qux.__class__</code>, corge.__dict__</code>. You will probably add breakpoints</a> here and there, to dive inside pdb</a>. Then start a REPL, then launch the program, then do some instrospection, then see what's going on. And finally, discover that the type is not what you were expecting. What about just reading the type?</h2> Let's be honest, as a reader, as the one who review millions of lines of code, you just want to be able to read what are the types. And I'm not talking about docstrings</a>. Everybody should document their programs and write docstrings. But let's face it, docstrings may be missing, misleading, out of date, not clear enough, etc.... Let's compare the 2 following shorts snippets: def get_description(self, tr): # What is tr ??? foo = self.get_foo() if foo != "Bar": return tr(foo) # ah, tr is a function. What's returned by tr? return "Bar" # The method seems to return a string </code></pre> Imagine having to find answers to those questions many times in a day? def get_description(self, tr: rt.TranslationFunction) -> Optional[str]: foo = self.get_foo() if foo != "Bar": return tr(foo) return "Bar" </code></pre> You see that in the second snippet, all the important informations are on the function header. You don't even read the function body and you already know that: "get_description is a method that takes a function as its only parameter and return a string or nothing". That's a game changer when you're a maintainer. It's not a surprise, all</a> the</a> big</a> companies</a>, the ones with large software systems, are already using type annotations. Too verbose and confusing to read?</h2> Adding type annotation can quickly lead to a source code that is too verbose and disgusting to read. The following is something I've written for RQL. I think I can do better (define type aliases, use stub files, use dataclasses, refactor, ...) class ScopeNode(BaseNode): def __init__(self): self.defined_vars: Dict[str, "rql.nodes.Variable"] = {} self.with_: List["rql.nodes.SubQuery"] = [] self.solutions: rt.SolutionsList = [] self._varmaker = None self.where: Optional["rql.base.Node"] = None self.having: Iterable["rql.base.Node"] = () self.schema: Optional[Any] = None self.aliases: Dict[str, "rql.nodes.ColumnAlias"] = {} </code></pre> Stub files?</h2> If you want to keep your source code free of type annotations, but still want to use type annotations, you can write stub files</a>. A stub file is just a file that will contain all your variables, function signatures, annotated. Function bodies in stub files are just a single ellipsis. Another really short example: # foo.py def get_description(self, tr): foo = self.get_foo() if foo != "Bar": return tr(foo) return "Bar" </code></pre> # foo.pyi <--- notice the extension used for stub files def get_description(self, tr: rt.TranslationFunction) -> Optional[str]: ... </code></pre> I want type annotations, what's next?</h2> Mypy</a> is a static type checker for Python. Let's see what Mypy can to for us. 1. Generating type annotations:</h3> Let's suppose you want to add type annotations to your code base. How do you proceed? Mypy includes the stubgen</a> tool that can automatically generate stub files. For example, here</a> is an example of a .pyi</code> file generated by the stubgen tool for the dateparser</a> package. from typing import Any, Optional def parse( date_string: Any, date_formats: Optional[Any] = ..., languages: Optional[Any] = ..., locales: Optional[Any] = ..., region: Optional[Any] = ..., settings: Optional[Any] = ...): ... </code></pre> Keep in mind, most types annotations will default to Any</code>. You'll need to update pyi</code> files and write more precise types (We agree that Any</code> everywhere is not really useful). Here</a> is an example of the same stub file with more precise type annotations: import datetime import sys from typing import Set, Tuple from dateparser.date import DateDataParser if sys.version_info >= (3, 8): from typing import Literal, TypedDict else: from typing_extensions import Literal, TypedDict __version__: str _default_parser: DateDataParser _Part = Literal["day", "month", "year"] _ParserKind = Literal["timestamp", "relative-time", "custom-formats", "absolute-time", "no-spaces-time"] class _Settings(TypedDict, total=False): DATE_ORDER: str PREFER_LOCALE_DATE_ORDER: bool TIMEZONE: str TO_TIMEZONE: str RETURN_AS_TIMEZONE_AWARE: bool PREFER_DAY_OF_MONTH: Literal["current", "first", "last"] PREFER_DATES_FROM: Literal["current_period", "future", "past"] RELATIVE_BASE: datetime.datetime STRICT_PARSING: bool REQUIRE_PARTS: list[_Part] SKIP_TOKENS: list[str] NORMALIZE: bool RETURN_TIME_AS_PERIOD: bool PARSERS: list[_ParserKind] def parse( date_string: str, date_formats: list[str] | Tuple[str] | Set[str] | None = ..., languages: list[str] | Tuple[str] | Set[str] | None = ..., locales: list[str] | Tuple[str] | Set[str] | None = ..., region: str | None = ..., settings: _Settings | None = ..., ) -> datetime.datetime | None: ... </code></pre> Do we agree that the : parse function parameters are more precise. </li> settings parameter is annotated with a more detailed type.2</a></li> returned value is more precise. </li> refactoring of this function will be easier.</li> </ul> 2. Type checking:</h3> Before running your tests, before running your program, Mypy can detect some types of bugs: I pretend foo</code> should return a string</code>, yet, there are cases where foo does return None</code>. That should be fixed. Notice how, the program is running fine. Mypy will type check the program and tell you what's wrong. Mypy won't stop you from running the program. Conclusion:</h2> Most people, including myself, don't work on projects of this size (4 million lines of Python code</a>), and hence have not to deal with the constraints of such a big code base. Most people can perfectly write their Python programs without using type annotations. Mypy is just another tool inside your toolbox. I just want you to try, that will certainly make you a better software developer. In the next article</a>, I will talk a little bit more about the actual process I've followed for adding type annotations to the RQL package. To learn more about this topic:</h2> Python typing</a></li> What Python can learn from Haskell</a></li> Type hinting (and mypy)</a></li> </ul> ^1Which, by the way, is where you'll find more interesting answers than what's inside the documentation, imho. 2</a>: Yes. I won't deny, there is suddenly more lines to read </div> Elephant in the room (part 1) 2021-05-25T03:40:17+00:00 I'm living in a Continent where (it sounds like) nobody really cares about software freedom, data privacy, surveillance. I guess, people here are simply too busy trying to stay alive</a>, not enough time to think about FOSS. Or maybe they're simply confused, maybe they simply don't know. The topic is wide, it's easy to digress, and I'm not sure about the best way to tackle it (consider this to be a first draft, updates will come). Let's say it's just an attempt, an effort, to at least address the issue. Okay. Let me put you in the context... Will you be okay with anybody trying to:</h1> Fix any of your furnitures.</li> Take you anywhere you want.</li> Offer you any food you want.</li> Pay for all your phone, sms, mail bills.</li> Let you watch any movie, any film, any show.</li> Alert you about something every 5mn.</li> </ul> And in exchange the same person:</h1> Won't ask you for any money, nothing, he'll just keep a tiny tiny list of all: the furnitures in your house?</li> the places you visit?</li> the foods you eat?</li> the conversations you had?</li> the movies you watched?</li> the time he notified you? The reason he notified you?</li> </ul> Do we agree that you won't be okay?</h1> In standard conditions for temperature and pressure, you won't be okay with anybody offering you the deal above because (among many other reasons): you don't want anybody to access your private space</li> you want to keep a certain level of autonomy, independance.</li> you want to preserve your physical integrity.</li> </ul> Interestingly:</h1> It sounds like, nobody want me to: put a webcam and a microphone in their house. </li> have the key to their house. </li> follow them. </li> take picture of them or recording them. </li> </ul> Yet, most of the same people are using all kinds of "privacy invading" technologies on a daily basis: Siri</a>, Alexa</a>, Google</a>. I won't even talk about smart TVs</a>, nor Facebook</a>. Any reasons that let us give our datas to the Tech Giants?</h1> People use privacy invading software for maaaaany reasons that can be understood. There is the "I need to figure out what to eat today" reason. Present all over the world, especially for the poorer people. Even more true in the Black continent. There is the fact that implications (of using privacy invading software) are invisible, not obvious. While the benefits are immediate, concrete, observable. Terms of Service are, most of the time, confusing</a>, difficult</a>, blurry</a>. There is also the "I use it because people around me are using it" reason. People are moved by those around</a> them and wish to fit in as quickly as possible, as simply as possible. Henrik Ibsen</a> said "A thousand words leave not the same deep impression as does a single deed". And I definitely agree with him: </center> Source</a> Do you know that:</h1> Your right to physical integrity was already affirmed in 1222, inside the Manden Charter</a>, article 5:</li> </ol> Everybody has a right to life and to the preservation of physical integrity. [...] </blockquote> Your right to privacy was affirmed in 1948 by the Universal Declaration of Human Rights</a>, article 12:</li> </ol> No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks. </blockquote> Within the European Convention on Human Rights</a>, in the article 8, we can read:</li> </ol> Right to respect for private and family life. Everyone has the right to respect for his private and family life, his home and his correspondence. </li> There shall be no interference by a public authority with the exercise of this right except such as is in accordance with the law and is necessary in a democratic society in the interests of national security, public safety or the economic well-being of the country, for the prevention of disorder or crime, for the protection of health or morals, or for the protection of the rights and freedoms of others. </li> </ol> </blockquote> The U.S. constitution</a> Fourth Amendment</a> states that:</li> </ol> The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized. </blockquote> The point seems to be: Every human being has a right to privacy, and this right should not be violated. What is privacy?</h1> I was not able to find a fixed, simple, perfect, static definition of privacy</a>. The definition may vary from one country to another, from one constitution to another, from one period of time to another. But I guess we can agree that privacy is any information that intrudes into the intimacy of the individual. Right? Privacy is that thing which is essential to the protection of our dignity as Human Being. Right? Anything that is directly related to : your personal space, your house, your image, </li> your financial status, your medical status,</li> your religious, political, or even sexual orientations,</li> </ul> Privacy is the fundamental way to protect yourself Some of the solutions we can explore:</h1> How do we defend ourselves and our friends from privacy invading technologies? What can we do to safeguard our personal privacy. What are the ways we can use to better protect ourselves when going online? What are the alternatives to the privacy invading software? I'll talk about the solutions in the next article</a> (coming soon). No, I'm not:</h1> Saying that I've all the answers.</li> Saying that you should do as I say.</li> Asking you to close all your accounts. </li> Asking you to go back to the stone age.</li> Trying to make you affraid of technology.</li> Saying that I've found all the alternatives.</li> Saying that migrating from privacy invading software will be an easy task.</li> </ul> What I'm asking you:</h1> Completely shuting down all the Tech Giants won't be an easy task. It cannot be done within the snap of a finger (I wish it could</a>). Let's stay optimistic, there is still some actionable steps we can take to help: Ask yourself : Why are you using those platforms?</li> What are the services you really need?</li> Is there any alternatives? (Respecting your privacy).</li> </ul> </li> Be aware: of the risks, of the pros & cons. </li> that if it's free, you're probably the product</a></li> </ul> </li> Ask for more: Transparency to all those platforms.</li> Regulation & protection to your leaders, your governments.</li> </ul> </li> Spark conversation, everywhere.</li> </ul> Interesting links:</h1> This topic is too wide to be covered entirely in this tiny article. I really hope you're learned something. I strongly recommend the following links for a better exploration and understanding of the topic. The following is a list of resources you should download, browse & share: Enemy of the State</a></li> Killswitch</a></li> We are legion</a></li> Terms and conditions may apply</a></li> Citizenfour</a></li> Social dilemma</a></li> The great hack</a></li> Snowden</a></li> Nos données personnelles valent de l'or</a></li> A Cypherpunk's Manifesto</a></li> Macron trahit la « start-up nation », les gafam raflent la mise</a></li> </ul> Thanks:</h1> Special thanks</a> to Orbifx, the Nature's emissary</a>, and Travis Kalanick</a>. They both gave me interesting ideas to help me finish this article. Do not forget:</h1> </center> Django single file project. Again 2020-07-10T01:09:16+00:00 This article is a follow up on the one</a> I've written few years ago. Special shoutout and thanks and hugs and kudos to Javier Buzzi</a> for taking the time to improve</a> the original script. TL;DR: download old version</a> & download new version</a> VERY IMPORTANT DISCLAIMER</h2> The content associated with mlvin.xyz may be very offensive to some people. I've lost the ownership of mlvin.xyz a very long time ago, sad situation. I'm not anymore linked to this domain name, nor to the content hosted on this website. By the way F*** OVH. What are those improvements?</h2> Application label?</h3> Our models are not defined as in a classical Django application. So, each model should declare which app it belongs to. You can read more about app_label Meta option here</a>. The problem with the first script was duplication: class Author(models.Model): name = models.CharField(max_length=200) class Meta: app_label = APP_LABEL class Book(models.Model): author = models.ForeignKey(Author, related_name='books') title = models.CharField(max_length=400) class Meta: app_label = APP_LABEL </code></pre> We want to define app_label once and for all our models: # The current name of the file, which will be the name of our app APP_LABEL, _ = os.path.splitext(os.path.basename(os.path.abspath(__file__))) </code></pre> Then, we need to overwrite the Django function in charge of: looking for an application configuration</li> attach the model to the found/correct application configuration.</li> </ul> This function is named get_containing_app_config</a> def get_containing_app_config(module): if module == '__main__': # if inside single file project, # retrieve the application configuration using APP_LABEL variable return apps.get_app_config(APP_LABEL) # otherwise, do as you always do return apps._get_containing_app_config(module) # save the original function apps._get_containing_app_config = apps.get_containing_app_config # ask Django to use our custom function apps.get_containing_app_config = get_containing_app_config </code></pre> Migrations folder?</h3> There is a Django settings named MIGRATION_MODULE</a>. According to the documentation</a>: A dictionary specifying the package where migration modules can be found on a per-app basis. The default value of this setting is an empty dictionary, but the default package name for migration modules is ``migrations``. </blockquote> In the first version of the script, all the migrations were put in a folder named migrations</code>. With the following configuration, we can: name the migration folder how we want.</li> make sure that we have one migration folder per application.</li> </ul> # Where migrations folder need to be created APP_MIGRATION_MODULE = '%s_migrations' % APP_LABEL APP_MIGRATION_PATH = os.path.join(BASE_DIR, APP_MIGRATION_MODULE) # Create the migration folder # and a __init__.py file if necessary: if not os.path.exists(APP_MIGRATION_PATH): os.makedirs(APP_MIGRATION_PATH) open(os.path.join(APP_MIGRATION_PATH, '__init__.py'), 'w').close() </code></pre> We also took the time to update the Django settings, somewhere inside the call to settings.configure function</code>, you'll find: # Django needs to be told where is the migration module MIGRATION_MODULES={APP_LABEL: APP_MIGRATION_MODULE}, </code></pre> Not so fake modules?</h3> A correctly configured Django application should normally have a model modules</code>, a urls modules</code> and a tests module</code>. Python provides us with an interesting class</a>: types.ModuleType</code>, that will help us create a module, you guessed it: # Used so you can do 'from <name of file>.models import *' models_module = ModuleType('%s.models' % (APP_LABEL)) tests_module = ModuleType('%s.tests' % (APP_LABEL)) urls_module = ModuleType('%s.urls' % (APP_LABEL)) # we created the urlpatterns list earlier, # let's attach this variable to the urls_module: urls_module.urlpatterns = urlpatterns # A model module needs to have classes that inherits from models.Model # let's correctly add all our models we defined earlier # as attributes to the models_module: for variable_name, value in list(locals().items()): # We are only interested in models if inspect.isclass(value) and issubclass(value, models.Model): setattr(models_module, variable_name, value) # Setup the fake modules sys.modules[models_module.__name__] = models_module sys.modules[urls_module.__name__] = urls_module sys.modules[APP_LABEL].models = models_module sys.modules[APP_LABEL].urls = urls_module </code></pre> Going further?</h3> I really hope you've learned something, I did \o/. You may want to browse some of the gists kept by Javier Buzzi</a>. You may also be really interested by the work done for django-micro</a> a lightweight wrapper around Django that turns it into a microframework. If you have any kind of interesting improvements, advices, suggestions, please send me an email, or post a comment on Mastodon</a>. Still not took the time to install a comment system, shame on me. o/ Free and Open Source Software 2017-04-04T00:00:00+00:00 I'm often asked why do I prefer to use free software in places (Dakar, Libreville, Kinshasa, ...), where basically every human being is using closed source and proprietary software? Here are some of the reasons. Not being able to pay for a license</h3> Learning informatics was equivalent to installing a bunch of software requiring me to pay for a license. And coming from a very modest family, paying for a license was not an option. I've never really been able to explain to my parent that I need to pay, via internet, using a credit card</li> a company located somewhere in the world, I don't even know where</li> to get the permission to use such an abstract thing like a software</li> for some time, then I'll have to pay again, to renew the permission</li> </ul> Tired of looking for the crack</h3> Not being able to pay for the license, the remaining choice was to look for the crack. that was the most easy way to get the software you want to use. But in the mean time, That was a very difficult task, always browsing the internet, looking for the product keys, the key generator, the patch, the full activator, etc... It was time consuming, and boring as editors are always trying to make it more and more difficult. Trying to be a good citizen</h3> Even if they should, most people often do not feel bad about using cracked software. After all, there were no physical harm, no shoplifting, no unlawful entry into a building, no indiscriminate taking of goods. Why feeling bad about it? Theft, from Wikipedia</a> In common usage, a theft is the illegal taking of another person's property or services without that person's permission or consent with the intent to deprive the rightful owner of it. </blockquote> Property, from Wikepedia</a> Important widely recognized types of property include real property (the combination of land and any improvements to or on the land), personal property (physical possessions belonging to a person), private property (property owned by legal persons, business entities or individual natural persons), public property (state owned or publicly owned and available possessions) and intellectual property (exclusive rights over artistic creations, inventions, etc.), although the last is not always as widely recognized or enforced. </blockquote> Above are the main reasons why I chose free and open source software. You can actually close this page, and go back to your awesome life, **thanks for reading**. But there is more. The free and open source software philosophy promotes values that go beyond software</h3> It's about the right to own an item, to use it for any purpose</li> It's about the right to study how this item is made</li> It's about the right to customize this item</li> It's about the right to share the item modified or not, with whoever you want</li> </ul> Fine, can we have some real life examples please?</h3> From Software Heritage</a> Software controls the electronic equipment embedded in the machines we use to travel, communicate, trade and exchange. Software is just crucial for the proper operation of every large organizations. Software has enabled the emergence of new forms of social and political organizations. Software is essential to understand our society. </blockquote> Maybe it's not perceptible enough, but free and open source software is already everywhere</a>. Here you are: Aerospace: Open Source Software projects from NASA</a></li> Automotive: AGL</a> group, including lots of automakers</a></li> Robotics: Open Source Robotics Foundation</a></li> Healthcare: Open-EMR</a>, OpenMRS</a></li> Education: Open Sankoré</a>, an interactive whiteboard software.</li> Films: Sintel</a>, and many other open movies by Blender Foundation</a>.</li> </ul> It is no more about software, it is about **openess and freedom**. The topic is actually very huge and deep, I've not even mentioned privacy, nor surveillance. I obviously won't be able to cover the whole thing. For those who want to go further, I strongly recommands the following links:</h3> Threats of data mining</a>.</li> Freedom in the cloud</a></li> Right to fix their own dang tractors</a></li> Why you should become an open source sustainer</a></li> Pourquoi l'open source</a> (French) </li> </ul> Threads 2017-03-08T04:30:25+00:00 I've written something about processes</a> Let's write some notes about threads. This is not really an introduction to threads. It's more like a little bit of introspection, so we can have an interesting perspective of what are threads. What's really a thread?</h4> Processes each have their own address space. Threads exist as subsets of a process. Threads are just multiple workers in the same virtual address space</a> all threads in a process share the same memory. They can also share open files and other resources. Within that VAS, each thread has its own ID, its own stack, its own program counter, its own independent flow of control, its own registers set. A thread is just a context of execution. Image from mooneegee.blogspot.sn</a> Next is a c script spawning two posix threads: #define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <pthread.h> #include <unistd.h> // sleep #include <sys/syscall.h> #include <sys/types.h> void * say_hello(void* data){ char *str = (char*)data; int howmany = 4; int r = (rand() % 20); pid_t thread_id = syscall(__NR_gettid); while(howmany > 0){ sleep(r); howmany = howmany - 1; printf("%s, ",str); printf("from thread %u and process %u\n", thread_id, getpid()); } return NULL; } int main(void){ pthread_t t1,t2; if (getpid() < 0) { perror("unable to get pid"); } else { printf("The main process id is %d\n", getpid()); } pthread_create(&t1,NULL,say_hello,"howdy"); pthread_create(&t2,NULL,say_hello,"hello"); pthread_join(t1,NULL); pthread_join(t2,NULL); return 0; } </code></pre> If we strace</a> the execution of our c script: $ strace -f ./bin/how_to_thread </code></pre> The output will display lots of informations. And for what we are doing, the most interesting lines will look like this: clone(child_stack=0x7f30fa9b8ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f30fa9b99d0, tls=0x7f30fa9b9700, child_tidptr=0x7f30fa9b99d0) = 7450 Process 7450 attached [pid 7449] clone(child_stack=0x7f30fa1b7ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f30fa1b89d0, tls=0x7f30fa1b8700, child_tidptr=0x7f30fa1b89d0) = 7451 </code></pre> The clone() system call:</h4> Processes</a> are created with the fork()</a> system call. However, there is a separate system call, named clone()</a> which is used for creating threads. It works like fork(), but it accepts a number of flags for adjusting its behavior so the child can share some parts of the parent's execution context. Our c script made a call to clone() twice. And looking at some of the flags that have been passed, we can see that: the caller and the child process: CLONE_VM: run in the same memory space.</li> CLONE_FS: share the same filesystem information.</li> CLONE_FILES: share the same file descriptor table.</li> CLONE_THREAD: are placed in the same thread group.</li> </ul> Another interesting arguments passed to clone system call are child_stack</code> and tls</code>. From the man page, we can read somewhere</a>: The child_stack argument specifies the location of the stack used by the child process. </blockquote> And in another part</a> of the man page: set a thread local storage (TLS) area </blockquote> Basically, we're allocating a personal stack to each of our threads. But, we also give each thread, an access to a region of memory which is not shared with all other threads. Like one may do with Python using the threading.local</a> class. GDB to the rescue</h4> Let's have a confirmation using GDB</a>: $ gdb ./bin/how_to_thread </code></pre> First, we set two breakpoints, one at the start of the main function, another one at the start of the say_hello function: Then we start the debugged program: r</code>, and list the mapped memory regions: So far, we see only one stack segment, the one used by the main process, that is because our threads are not yet created. Let's continue a little bit (twice) with c</code>, and you should be able to see something like this: This time, we can see three stacks in the same virtual address space. The ones used by our threads are [stack:15074]</code> and [stack:15075]</code>. GDB to inspect each thread's stack: LWP is an acronym and stands for Light Weight Process. Next to LWP is the thread ID assigned by the kernel to this thread. We see that each thread has its own stack, and inside each stack, the progress is different, while thread 3 is entering the say_hello function, thread 2 is in nanosleep. By the way nanosleep is also a system call</a>. Here we are, a thread is indeed just a context of execution. I won't be able to cover the whole topic as it's not an easy and small one. So you may find really interesting the following links: Pthreads</a></li> Making Linux safe for pthreads</a></li> Context switch</a></li> Semaphores</a></li> The real reason not to share a mutable state</a></li> </ul> Django single file project 2017-01-12T17:17:17+00:00 Disclaimer?</h4> The main purpose of this post is learning. Like any tool we should explore and find ways to bend it until it breaks. You should normally not use those techniques in real life projects right? For this post, I'm assuming you're using Django 1.10.4, and Python3.6. Single file?</h4> Before reading Lightweight Django</a>, I never tought it was possible to build a Django project in a single python file. For that kind of stuff, I usually think, Flask</a>, or Bottle</a>. And in fact, there are many others</a>. After reading the first chapter, I was a little bit disappointed. There were no Django models</a> involved in the shown example. So, I try to find out if it was possible: a single file Django project with at least one application containing one model. I found some interesting links on the topic. But either it was for a very old version of Django, either I was obliged to install another package besides Django. After few days reading others people code, this is what I've done: #!/usr/bin/env python import os import sys from django.conf import settings BASE_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path[0] = os.path.dirname(BASE_DIR) # the current folder name will also be our app APP_LABEL = os.path.basename(BASE_DIR) settings.configure( DEBUG=os.environ.get('DEBUG', 'on') == 'on', SECRET_KEY=os.environ.get('SECRET_KEY', os.urandom(32)), ALLOWED_HOSTS=os.environ.get('ALLOWED_HOSTS', 'localhost').split(','), ROOT_URLCONF=__name__, MIDDLEWARE=[ 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.locale.LocaleMiddleware', ], INSTALLED_APPS=[ APP_LABEL, 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', ], STATIC_URL='/static/', STATICFILES_DIRS=[ os.path.join(BASE_DIR, "static"), ], STATIC_ROOT=os.path.join(BASE_DIR, "static_root"), MEDIA_ROOT=os.path.join(BASE_DIR, "media"), MEDIA_URL='/media/', TEMPLATES=[ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(BASE_DIR, "templates"),], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.template.context_processors.tz', 'django.contrib.messages.context_processors.messages', ], }, }, ], DATABASES={ 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), } }, REST_FRAMEWORK={ 'DEFAULT_PERMISSION_CLASSES': [ 'rest_framework.permissions.IsAdminUser', ], 'PAGE_SIZE': 10 } ) import django django.setup() from django.db import models from django.contrib import admin from django.db import models # Create your models here. class Author(models.Model): name = models.CharField(max_length=200) class Meta: app_label = APP_LABEL class Book(models.Model): author = models.ForeignKey(Author, related_name='books') title = models.CharField(max_length=400) class Meta: app_label = APP_LABEL admin.site.register(Book) admin.site.register(Author) admin.autodiscover() from rest_framework import serializers class BookSerializer(serializers.ModelSerializer): class Meta: model = Book fields = '__all__' from rest_framework import viewsets class BooksViewSet(viewsets.ReadOnlyModelViewSet): queryset = Book.objects.all() serializer_class = BookSerializer from django.conf.urls import url, include from rest_framework import routers from django.http import HttpResponse from django.contrib import admin router = routers.DefaultRouter() router.register(r'books', BooksViewSet) def index(request): return HttpResponse("Hello") urlpatterns = [ url(r'^admin/', admin.site.urls), url(r'^$', index, name='homepage'), url(r'^api/', include(router.urls)), url(r'^api-auth/', include('rest_framework.urls',\ namespace='rest_framework')) ] from django.core.wsgi import get_wsgi_application if __name__ == "__main__": from django.core.management import execute_from_command_line execute_from_command_line(sys.argv) else: get_wsgi_application() </code></pre> Everything related to Django REST Framework</a> is not mandatory. I was looking for interesting links</a> over the internet, and I found this question on StackOverflow</a>. I saw this as an opportunity to kill two birds with one stone. The above script was saved in a file named single.py</code> in a folder named lwdj</code>. But you can do as you feel, as soon as you respect the name of the file and the name of the current folder. The rest is known story: Make the migrations for our application: (dj) ~/.../how_to_django/lwdj (master *%) ⤷ python single.py makemigrations lwdj Migrations for 'lwdj': migrations/0001_initial.py: - Create model Author - Create model Book (dj) ~/.../how_to_django/lwdj (master *%) ⤷ </code></pre> Run the migrations: (dj) ~/.../how_to_django/lwdj (master *%) ⤷ python single.py migrate Operations to perform: Apply all migrations: admin, auth, contenttypes, lwdj, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying lwdj.0001_initial... OK Applying sessions.0001_initial... OK (dj) ~/.../how_to_django/lwdj (master *%) </code></pre> Create a super user and run the server: (dj) ~/.../how_to_django/lwdj (master *%) ⤷ python single.py createsuperuser Username (leave blank to use 'nsukami'): Email address: nsukami@gmail.com Password: Password (again): Superuser created successfully. </code></pre> (dj) ~/.../how_to_django/lwdj (master *%) ⤷ python single.py runserver Performing system checks... System check identified no issues (0 silenced). January 12, 2017 - 18:52:57 Django version 1.10.4, using settings None Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C. </code></pre> So, what have we learned here?</h4> Yes, that is not how you are supposed to use Django. But we were able to learn few things. Projects and applications</h4> Somewhere in the Django documentation related to projects and applications</a>, we can read: There's no restriction that a project package can't also be considered an application and have models, etc. (which would require adding it to INSTALLED_APPS). </blockquote> We can double check using the tree</code> command: (dj) ~/.../how_to_django/lwdj (master *%) ⤷ tree . ├── activate -> /home/nsukami/envs/dj/bin/activate ├── db.sqlite3 ├── Makefile ├── media ├── migrations │ ├── 0001_initial.py │ ├── __init__.py │ └── __pycache__ │ └── __init__.cpython-36.pyc ├── single.py ├── static │ └── site.css ├── static_root └── templates └── home.html </code></pre> The django.setup() method</h4> Reading The Fabulous Manual</a>, we learn that, this method configures Django by loading the settings, but also: When Django starts, django.setup() is responsible for populating the application registry. </blockquote> Let's start a shell using the command python single.py shell</code>, and let's check the application registry: (dj) >>> from django.apps import apps (dj) >>> apps.ready True (dj) >>> (dj) >>> (dj) >>> apps.get_app_configs() odict_values([<AppConfig: lwdj>, <AdminConfig: admin>, <AuthConfig: auth>, <ContentTypesConfig: contenttypes>, <SessionsConfig: sessions>, <MessagesConfig: messages>, <StaticFilesConfig: staticfiles>, <AppConfig: rest_framework>]) (dj) >>> </code></pre> (dj) >>> c = apps.get_app_config('lwdj') (dj) >>> (dj) >>> c.module <module 'lwdj' (namespace)> (dj) >>> (dj) >>> c.path '/home/nsukami/GITHUB/how_tos/how_to_django/lwdj' (dj) >>> (dj) >>> c.verbose_name 'Lwdj' (dj) >>> (dj) >>> c.name 'lwdj' (dj) >>> c.label 'lwdj' (dj) >>> (dj) >>> for m in c.models: print(m) ... author book (dj) >>> </code></pre> Neat! We also read</a> that: At this stage, your code shouldn't import any models! </blockquote> This is exactly why our models, and everything related to them, do not appear before those 2 lines of code. import django django.setup() </code></pre> Doing otherwise, raises the AppRegistryNotReady</code> exception: (dj) ~/.../how_to_django/lwdj (master *%) ⤷ python single.py runserver Traceback (most recent call last): File "single.py", line 90, in <module> class Author(models.Model): File "/home/nsukami/envs/dj/lib/python3.6/site-packages/django/db/models/base.py", line 105, in __new__ app_config = apps.get_containing_app_config(module) File "/home/nsukami/envs/dj/lib/python3.6/site-packages/django/apps/registry.py", line 237, in get_containing_app_config self.check_apps_ready() File "/home/nsukami/envs/dj/lib/python3.6/site-packages/django/apps/registry.py", line 124, in check_apps_ready raise AppRegistryNotReady("Apps aren't loaded yet.") django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. (dj) ~/.../how_to_django/lwdj (master *%) </code></pre> Installed applications</h4> I have not yet fully understood this part. APP_LABEL</code> is actually containing the name of the current directory. By putting APP_LABEL</code> inside INSTALLED_APPS</code>, Django was able to find all the Django models existing inside the current directory: BASE_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path[0] = os.path.dirname(BASE_DIR) APP_LABEL = os.path.basename(BASE_DIR) {{< /highlight >}} {{< highlight python >}} INSTALLED_APPS=[ APP_LABEL, 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', ], </code></pre> App label</h4> All our models were created outside of a models.py</code> file. They're not even in a real Django application. So we were obliged to tell them which application they belongs to. This is done using the app_label</a> attribute of the internal class Meta</code> of Django models. {{< highlight python >}} class Author(models.Model): name = models.CharField(max_length=200) class Meta: app_label = APP_LABEL class Book(models.Model): author = models.ForeignKey(Author, related_name='books') title = models.CharField(max_length=400) class Meta: app_label = APP_LABEL {{< /highlight >}} Really hope you learnt something. Also, I would not be able to write this post, without following the StackTrace leaved by the people who were there before me. A huge thanks to them: Créer une application django minimaliste</a></li> Django Models without Apps OR Everything Django Truly in a Single File</a></li> Django as microframework</a></li> </ul> Make 2016-11-25T00:00:00+00:00 if you have some complex and esoteric shell commands that you have to run on a regular basis, it can be convenient to just package them up somewhere, so you don't have to remember them or type them out every time. Why, I don't use Makefiles more often? Even though I know I should. One of the reason may be because, as an Emacs user, automating simple tasks is funny using Emacs Lisp</a>. Following are simple elisp functions I use to compile a C file, nothing extraordinary: (defun my-c-save-compile () "Save, and compile c file." (interactive) (save-buffer) (compile (format "cc -Wall -Wextra -pedantic -std=c11 -g %s -o ./bin/%s" (buffer-file-name) (file-name-base (buffer-file-name))))) (defun my-c-run () "Run c code." (interactive) (compile (format "./bin/%s" (file-name-base (buffer-file-name))))) </code></pre> Another reason I don't use Makefiles more often could be, the Bourne Again Shell</a>. I find it easier to quickly write shell aliases for repetitive tasks. The following is some aliases I sometimes use whenever I'm doing Django</a> development: #!/bin/sh # my weird aliases alias djo_runserver="python manage.py runserver" alias djo_shell="python manage.py shell" alias djo_shellp="python manage.py shell_plus" alias djo_make="python manage.py makemigrations" alias djo_mig="python manage.py migrate" alias djo_csu="python manage.py createsuperuser" alias djo_smtpd="python -m smtpd -n -c DebuggingServer localhost:1025" </code></pre> Make do have a reputation for being complex. Indeed, it can be very esoteric</a>. The full manual</a> is a 183 pages. Fortunately, you can ignore most of this, Make can be used in a very simple and incredibly useful way. The following, is a Makefile I sometimes use in replacement of the shell aliases above: # Makefile for (the project) # # Nsukami # November 2016 SHELL := /bin/sh PROJECT := tas PYTHON_BIN := $(VIRTUAL_ENV)/bin DEV_SETTINGS = $(PROJECT).settings.dev PROD_SETTINGS = $(PROJECT).settings.prod TEST_SETTINGS = $(PROJECT).settings.test BASE_DIR = $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST)))) PMP = python manage.py # if not on prod server, use dev settings ifneq ($(shell uname -n),phobos) SETTINGS = $(DEV_SETTINGS) else SETTINGS = $(PROD_SETTINGS) endif # target: all - Default target. Does nothing. all: @echo "Nothing to do by default. Try 'make help'" # target: help - Show list of targets + description help: @egrep "^# target:" [Mm]akefile # target: init - Install that a dev needs to get up and running. init: bootstrap.sh cd $(BASE_DIR) && sudo ./bootstrap.sh # target: clean - remove all useless files clean: clean-pyc clean-build # target: clean-pyc - remove all auto generated files clean-pyc: find . -name '*.pyc' -exec rm --force {} + find . -name '*.pyo' -exec rm --force {} + find . -name '*~' -exec rm --force {} + # target: clean-build - remove all auto generated folders clean-build: -rm --force --recursive build/ -rm --force --recursive dist/ -rm --force --recursive htmlcov -rm --force --recursive .coverage -rm --force --recursive *.egg-info # target: translate - calls the "makemessages" django command translate: cd $(BASE_DIR) && $(PMP) makemessages --settings=$(SETTINGS) -a # target: test - calls the "test" django command test: $(PMP) test --settings=$(TEST_SETTINGS) # target: run - calls the "runserver" django command run: $(PMP) runserver --settings=$(SETTINGS) # target: update - install (and update) pip requirements update: pip install -U -r $(BASE_DIR)/requirements/dev.txt # target: collect - calls the "collectstatic" django command collect: $(PMP) collectstatic --settings=$(SETTINGS) --noinput # target: rebuild - rebuild tables from models rebuild: $(PMP) makemigrations --settings=$(SETTINGS) $(PMP) migrate --settings=$(SETTINGS) # target: deploy - let make take care of deployment deploy: # Should I call fabric here? .ONESHELL: .PHONY: all help translate test clean update collect rebuild run clean-pyc clean-build clean init </code></pre> Finally, the other reason, and probably the main reason I don't use Makefiles more often, it's because of Fabric</a>. Following is a fabfile more or less doing the same thing as the Makefile above: #!/usr/bin/env python # -*- coding: utf-8 -*- # """ fabric commands """ from fabric.api import env, sudo, cd, local, task, run import fabric.contrib.project as project import os # Local path configuration (can be absolute or relative to fabfile) env.hosts = ['localhost'] env.use_ssh_config = True env.ssh_config_path = '/home/nsukami/.ssh/config' env.owner = 'nsukami' PROJECT = "tas" DEV_SETTINGS = PROJECT + ".settings.dev" PROD_SETTINGS = PROJECT + ".settings.prod" TEST_SETTINGS = PROJECT + ".settings.test" BASE_DIR = os.path.dirname(os.path.abspath(__file__)) PMP = "python manage.py " def uname(): """ Prints information about the host. """ local("uname -n") if uname() is not "phobos": SETTINGS = DEV_SETTINGS else: SETTINGS = PROD_SETTINGS @task def init(): "init - Install that a dev needs to get up and running." with cd(BASE_DIR): sudo("./bootstrap.sh") @task def clean(): "clean - remove all useless files" clean_pyc() clean_build() def clean_pyc(): "# clean-pyc - remove all auto generated files" local("find . -name '*.pyc' -exec rm --force {} +") local("find . -name '*.pyo' -exec rm --force {} +") local("find . -name '*~' -exec rm --force {} +") def clean_build(): "clean-build - remove all auto generated folders" local("rm --force --recursive build") local("rm --force --recursive dist") local("rm --force --recursive htmlcov") local("rm --force --recursive .coverage") local("rm --force --recursive *.egg-info") @task def translate(): "translate - calls the 'makemessages' django command" with cd(BASE_DIR): cmd = PMP + "makemessages --settings={} -a".format(SETTINGS) local(cmd) @task def test(): "test - calls the 'test' django command" cmd = PMP + "test --settings={}".format(SETTINGS) local(cmd) @task def runserver(): "run - calls the 'runserver' django command" cmd = PMP + "runserver --settings={}".format(SETTINGS) local(cmd) @task def update(): "update - install (and update) pip requirements" cmd = "pip install -U -r {}/requirements/dev.txt".format(BASE_DIR) local(cmd) </code></pre> We see that Fabric can be used as a build tool even though it isn't one. It is easy to read and write, especially for a Python developer. The drawback may be, it does require Python to run. Seriously, who cares? New year resolution: using Makefiles more often. More on the topic: Make</a></li> Make alternatives</a></li> List of build automation software</a></li> </ul> How to create processes 2016-08-22T03:40:17+00:00 We've seen processes</a>, We've said lots of things about processes, but we did not talk about process creation. Let's see how to create a process in C, in Golang</a>, and in Rustlang</a> Disclaimer: I'm not doing a comparison to find out which one is better than the other. I just want to explore how the same task can be done using different approaches. Fork and Exec:</h3> UNIX processes are created using the fork() system call</a>. Fork is the primary method of process creation on Unix-like operating systems. Fork is the system call that the parent process uses to "divide" itself, to "fork" into two identical processes. The child process will be the exact copy of the parent except for the return value. The exec()</a> system call and its family of functions are almost always used with fork(). The exec() syscall and the like overwrite the current [stackframe]({{< ref on-processes >}})with the name of the application passed to it. C:</h4> In C, there is nothing special to do, contrary to Golang and Rustlang. You just really call fork(), then exec(), simple and direct. The following, is an example written in C: #include <stdio.h> #include <sys/types.h> #include <unistd.h> #include <stdlib.h> #include <sys/wait.h> #include <string.h> void fork_and_execute(); int main() { fork_and_execute(); return 0; } void fork_and_execute() { int pid, status; pid = fork(); // if pid equals 0, if(pid == 0){ // we're dealing with the child process printf("I am the child process, my pid is %d\n", getpid()); // the program we want to execute char *cmd[5] = {"ls", "-a", "-l", "-h", '\0'}; // execvp will search for ls on $PATH if (execvp(*cmd, cmd) < 0){ // catch any error that may occur during execution printf("*** ERROR: exec failed\n"); perror(*cmd); exit(EXIT_FAILURE); } } // returned pid is -1, fork operation failed if(pid < 0){ printf("*** ERROR: forking child process failed\n"); perror("fork"); exit(EXIT_FAILURE); } // The wait system-call blocks the parent process // and waits for the child-process to end. while (wait(&status) != pid){} } </code></pre> Go:</h4> In Go, it is recommended to use the exec</a> package. It runs external commands, and wraps os.StartProcess</a> function for you. Following, is the previous example, this time written in Golang: package main import "syscall" import "os" import "os/exec" func main() { // check if ls command exists in the PATH binary, lookErr := exec.LookPath("ls") if lookErr != nil { panic(lookErr) } // the program we want to execute args := []string{"ls", "-a", "-l", "-h"} // fork new process and execute our program execErr := syscall.Exec(binary, args, os.Environ()) // catch error if any if execErr != nil { panic(execErr) } } </code></pre> Rust:</h4> In Rust, we have std::process::Command</a>. It is is a type that acts as a process builder and provides fine-grained control over how the new process should be spawned. If we browse the source code</a>, we can read beginning line 521: Constructs a new Command</code> for launching the program at </blockquote> path program</code>, with the following default configuration: </blockquote> </blockquote> No arguments to the program</li> </ul> </blockquote> Inherit the current process's environment</li> </ul> </blockquote> Inherit the current process's working directory</li> </ul> </blockquote> Inherit stdin/stdout/stderr for spawn</code> or status</code>, but create pipes for output</code></li> </ul> </blockquote> Basically, it behave a little bit like a call to fork(), followed by a call to exec(). Let's rewrite the previous example in Rust: use std::process::Command; fn main() { let output = Command::new("ls") .arg("-a") .arg("-l") .arg("-h") .output() .expect("ls command failed to start"); println!("stdout: {}", String::from_utf8_lossy(&output.stdout)); } </code></pre> The output</a> function executes the command as a child process, waiting for it to finish and collecting all of its output. By default, stdin, stdout and stderr are captured (and used to provide the resulting output). The expect</a> function is used for quick and dirty error handling. Strace, trace system calls and signals:</h3> Somewhere in the strace's manual page, we can read: Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system calls. </blockquote> If everything went normally during the execution of our scripts, we should see, in the strace's output, lots of system calls, the most important ones being a call to clone(), followed by a call to exec(): ... 4714 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fac5d28da90) = 4715 ... 7564 execve("/bin/ls", ["ls", "-a", "-l", "-h"], [/* 74 vars */] <unfinished ...> ... </code></pre> fork() was the original UNIX system call. Only used to create new processes, not threads. Also, it is portable. clone() is a new, versatile system call which can be used to create, depending on the passed option, a UNIX process, or a POSIX thread, or something in between, or something completely different. I invite you to Read The Fabulous Manual</a> about clone(). I don't really know if there is something else you can read to learn more on this topic: Fork</a> manual page.</li> Exec</a> manual page.</li> Clone</a> manual page.</li> Strace</a> manual page.</li> </ul> Processes 2016-08-09T03:40:17+00:00 As a Debian user, I often want to see the active processes, sometimes, I want to kill them. I know how to launch a new process, inside my Python scripts. But what is a process? A process is the execution context of a running program. When an executable program is read into memory by the kernel and executed, it becomes a process. Processes are dynamic entities, they are constantly changing as their machine code instructions are executed by the CPU. With the exception of the init process</code>, every process is the child of another process. And every process has the potential to be the parent of another process. The following is a small C program that will print the current process ID and the corresponding parent process ID: #include <stdio.h> #include <unistd.h> int main (){ char ch; printf ("The process ID is %d\n", (int) getpid ()); printf ("The parent process ID is %d\n", (int) getppid ()); while ( ch != '\r' && ch != '\n' ){ ch = getchar(); } return 0; } </code></pre> After execution, you should see something like: ~ +-> ./bin/ex1 The process ID is 11147 The parent process ID is 9552 </code></pre> </a> Virtual address space?</h3> A virtual address space, VAS</code>, is the set of ranges of virtual addresses available to a process. The virtual address space is what a process sees when it executes. A process works with and only sees VAS. The operating system handles mapping VAS to real storage so that this is invisible to the running program. We need VAS to prevent process A and process B from writing/reading in the same memory addresses. VAS is a way to achieve process isolation</a>. With VAS, several programs could write to memory location at the same address without stepping over each others results. </center> Image from tenouk.com</a> What's inside VAS?</h3> VAS</code>, is the set of ranges of virtual addresses available to a process, so what is inside a VAS? A typical memory representation of process consists of following sections: Stack: VAS section that is used for several related purposes, the main one being to keep track of the point to which each active subroutine should return control when it finishes executing. The return address is pushed onto the stack by the caller. When it finishes, the called subroutine, pops the return address off the stack and transfers control to that address. That section also stores temporary variables created by each function inside the program. The stack is a LIFO data structure, managed and optimized by the CPU quite closely. Every time a function declares a new variable, it is "pushed" onto the stack, you don't have to allocate the needed memory by hand. Then every time a function exits, all of the variables pushed onto the stack by that function, are freed for you. Once a stack variable is freed, that region of memory becomes available for other stack variables. </li> Heap: VAS section where dynamic memory allocation usually takes place. Unlike the Stack, this part of the VAS is not managed automatically for you, and is not as tightly managed by the CPU. You are in charge of allocating memory on the heap. You are also responsible for deallocating that memory once you don't need it any more. If you fail to do this, your program will have what is known as a memory leak. </li> Initialized Data Segment: Or simply, Data Segment. VAS section, which contains the global and static variables that are initialized by the programmer. The size of this segment is determined by the size of the values in the program's source code, and does not change at run time. </li> Uninitialized (bss) Data Segment: VAS section that contains all global and static variables that are initialized to zero or do not have explicit initialization in the programer's source code. </li> Text (code) Segment: VAS section that contains machine instructions of the program. Those machine instructions can be thought of as the text of a novel: It tells the story of what the program does. </li> </ol> How to do some introspection?</h3> How do we learn about the memory layout of a running process? How do we look at how program are laid out in memory? Directly from /proc/PID/maps: You can directly ask to the process information pseudo-filesystem: proc</a>. Let's see the output of cat /proc/6026/maps</code>. Here 6026</code> is the PID of a running process on my Debian: </li> </ol> 00400000-00401000 r-xp 00000000 08:05 3181468 /home/nsukami/bin/how_to_vms 00600000-00601000 rw-p 00000000 08:05 3181468 /home/nsukami/bin/how_to_vms 01698000-016b9000 rw-p 00000000 00:00 0 [heap] 7f747118a000-7f747132c000 r-xp 00000000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so 7f747132c000-7f747152b000 ---p 001a2000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so 7f747152b000-7f747152f000 r--p 001a1000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so 7f747152f000-7f7471531000 rw-p 001a5000 08:01 131521 /lib/x86_64-linux-gnu/libc-2.19.so 7f7471531000-7f7471535000 rw-p 00000000 00:00 0 7f7471535000-7f7471555000 r-xp 00000000 08:01 130924 /lib/x86_64-linux-gnu/ld-2.19.so 7f7471734000-7f7471737000 rw-p 00000000 00:00 0 7f7471752000-7f7471755000 rw-p 00000000 00:00 0 7f7471755000-7f7471756000 r--p 00020000 08:01 130924 /lib/x86_64-linux-gnu/ld-2.19.so 7f7471756000-7f7471757000 rw-p 00021000 08:01 130924 /lib/x86_64-linux-gnu/ld-2.19.so 7f7471757000-7f7471758000 rw-p 00000000 00:00 0 7ffc4d89b000-7ffc4d8bc000 rw-p 00000000 00:00 0 [stack] 7ffc4d99b000-7ffc4d99d000 r-xp 00000000 00:00 0 [vdso] 7ffc4d99d000-7ffc4d99f000 r--p 00000000 00:00 0 [vvar] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] </code></pre> The Heap segment and the Stack segment are labelled, easy. The addresses are displayed from the lowest `00400000`, to the highest `ffffffffff601000`. The first segment, `00400000-00401000`, is read-only and executable, so, this is the Code segment. We know that between the Code segment, and the Heap segment, there is the Data segment. If you don't want to browse the `/proc` directory, there is also the [pmap](http://linux.die.net/man/1/pmap) command you can use to report the memory map of a process. Using the GNU debugger: GDB, the awesome GNU Project debugger</a>, allows you, among others things, to see what is going on inside</code> a running process. For example, from the gdb prompt, you can type info proc mappings</code> to see the current address mapping:</li> </ol> process 8460 Mapped address spaces: Start Addr End Addr Size Offset objfile 0x400000 0x401000 0x1000 0x0 /home/nsukami/bin/how_to_vms 0x600000 0x601000 0x1000 0x0 /home/nsukami/bin/how_to_vms 0x601000 0x622000 0x21000 0x0 [heap] 0x7ffff7a31000 0x7ffff7bd3000 0x1a2000 0x0 /lib/x86_64-linux-gnu/libc-2.19.so 0x7ffff7bd3000 0x7ffff7dd2000 0x1ff000 0x1a2000 /lib/x86_64-linux-gnu/libc-2.19.so 0x7ffff7dd2000 0x7ffff7dd6000 0x4000 0x1a1000 /lib/x86_64-linux-gnu/libc-2.19.so 0x7ffff7dd6000 0x7ffff7dd8000 0x2000 0x1a5000 /lib/x86_64-linux-gnu/libc-2.19.so 0x7ffff7dd8000 0x7ffff7ddc000 0x4000 0x0 0x7ffff7ddc000 0x7ffff7dfc000 0x20000 0x0 /lib/x86_64-linux-gnu/ld-2.19.so 0x7ffff7fd7000 0x7ffff7fda000 0x3000 0x0 0x7ffff7ff5000 0x7ffff7ff8000 0x3000 0x0 0x7ffff7ff8000 0x7ffff7ffa000 0x2000 0x0 [vdso] 0x7ffff7ffa000 0x7ffff7ffc000 0x2000 0x0 [vvar] 0x7ffff7ffc000 0x7ffff7ffd000 0x1000 0x20000 /lib/x86_64-linux-gnu/ld-2.19.so 0x7ffff7ffd000 0x7ffff7ffe000 0x1000 0x21000 /lib/x86_64-linux-gnu/ld-2.19.so 0x7ffff7ffe000 0x7ffff7fff000 0x1000 0x0 0x7ffffffde000 0x7ffffffff000 0x21000 0x0 [stack] 0xffffffffff600000 0xffffffffff601000 0x1000 0x0 [vsyscall] </code></pre> And there is more to explore: (gdb) help info proc Show /proc process information about any running process. Specify any process id, or use the program being debugged by default. List of info proc subcommands: info proc all -- List all available /proc info info proc cmdline -- List command line arguments of the process info proc cwd -- List current working directory of the process info proc exe -- List absolute filename for executable of the process info proc mappings -- List of mapped memory regions info proc stat -- List process info from /proc/PID/stat info proc status -- List process info from /proc/PID/status Type "help info proc" followed by info proc subcommand name for full documentation. Type "apropos word" to search for commands related to "word". Command name abbreviations are allowed if unambiguous. (gdb) </code></pre> How to create a process?</h3> 1.fork(): The fork() system call is used to create processes. It takes no arguments and returns a process ID. The purpose of fork() is to create a new process, which becomes the child process of the caller: #include <stdio.h> #include <sys/types.h> #include <unistd.h> #include <stdlib.h> #include <sys/wait.h> #include <string.h> void execute(); int main() { execute(); return 0; } void execute() { int pid, status; if((pid = fork()) == 0){ char *cmd[5] = {"ls", "-a", "-l", "-h", '\0'}; if (execvp(*cmd, cmd) < 0){ printf("*** ERROR: exec failed\n"); perror(*cmd); exit(1); } }else{ printf("*** ERROR: forking child process failed\n"); perror("fork"); exit(EXIT_FAILURE); } //The parent executes the wait. while (wait(&status) != pid){} } </code></pre> I really would like to write more on this topic. Maybe in another post, otherwise the article will be too long, and I have the dishes to do. At least, we've seen that there is an interesting amount of informations behind what is called a process. I hope you've learned something, and if you want to know more:</h3> Processes</a></li> Proc</a></li> Proc man pages</a></li> The "vsyscall" and "vDSO" segments</a></li> GDB</a></li> The jealous process</a></li> </ul> Network namespaces 2016-07-08T00:00:00+00:00 I use LXC containers mainly as isolated environments when I'm writing code, but also when I just want to try the last version of a package without being obliged to break something on my current configuration. This is the first part of a serie about LXC containers. In this post, I'll share with you what I've learned about network namespaces. Everything here will be done under Debian Jessie, as root, I'm using Python 3.5 and I've pip installed pyroute2. To keep the schemas as simple as possible, I won't mention the interface lo</code> actually available inside the root namespace</code>. The root namespace</code> is the namespace available to you after a fresh boot of your Operating System. This is a small schema of my configuration showing you where we'll start: What we will do: Create a namespace</a>.</li> Create a virtual ethernet device pair</a>.</li> Assign one side of the veth pair to the namespace</a>.</li> Assign IP addresses to each side of the veth pair</a>.</li> Set the default route for our namespace</a>.</li> Attach the other side of the veth pair to a bridge</a>. </li> </ol> Network namespaces</h4> A namespace is a way of scoping a particular set of identifiers. You can use the same identifier multiple times in different namespaces. Most importantly, you can also restrict an identifier set visible to particular processes. Fact: the set of network interfaces and routing tables are shared across your entire Operating System. Network namespaces</a> change that fundamental assumption. With network namespaces, you can have different and separate instances of network interfaces and routing tables that operate independent of each other. Let's see how it works, using the Python package Pyroute2</a>: </a> >>> from pyroute2 import IPDB, IPRoute, NetNS, netns, NSPopen >>> from subprocess import Popen, PIPE >>> ipdb_namespace1 = IPDB(nl=NetNS('namespace1')) >>> nsp = NSPopen('namespace1', ['ip', 'addr'], stdout=PIPE) >>> for l in nsp.communicate()[0].split(b'\n'): ... print(l) ... </code></pre> b'1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default ' b' link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00' b'' </code></pre> We've just created a network namespace named namespace1</code>. The ip addr</code> command launched inside the namespace namespace1</code> tells us that there is only one interface, lo</code>. Now, we need a way to link our newly created namespace, to the root namespace</code>. To do that, virtual ethernet devices, veth pair</code>, need to be created and configured. A veth pair</code> works like a patch cable, connecting two sides. It consists of two virtual interfaces, one of them is assigned to the root network namespace, while the other lives within the network namespace, namespace1</code>: </a> >>> ipdb_root = IPDB() # root namespace >>> ipdb_root.create(ifname='v01', kind='veth', peer='vp01').commit() </code></pre> {'linkmode': 0, 'broadcast': 'ff:ff:ff:ff:ff:ff', 'num_rx_queues': 1, 'qdisc': 'noop', 'txqlen': 1000, 'ipaddr': [], 'group': 0, 'flags': 4098, 'neighbours': [], 'mtu': 1500, 'promiscuity': 0, 'vlans': [], 'address': 'ca:a0:d7:39:57:22', 'ipdb_scope': 'system', 'kind': 'veth', 'ipdb_priority': 0, 'ifi_type': 1, 'family': 0, 'carrier': 0, 'ports': [], 'peer': 'vp01', 'num_tx_queues': 1, 'index': 10, 'ifname': 'v01', 'carrier_changes': 1, 'operstate': 'DOWN'} >>> </code></pre> Veth pair device created. Both sides of the veth pair device are actually in the root namespace</code>. To link the root namespace</code> to our newly created namespace namespace1</code>, one side of the veth pair device, the peer, should be assigned to namespace1</code>: </a> >>> with ipdb_root.interfaces.vp01 as v: ... v.net_ns_fd = 'namespace1' ... >>> </code></pre> Now, let's give IP addresses to all of our interfaces, and set everybody up: </a> >>> with ipdb_root.interfaces.v01 as veth: ... veth.add_ip('192.168.1.36/24') ... veth.up() ... </code></pre> {'kind': 'veth', 'promiscuity': 0, 'group': 0, 'ipdb_scope': 'system', 'peer': 'vp01', 'linkmode': 0, 'ipdb_priority': 0, 'carrier_changes': 1, 'address': 'f2:2d:aa:9d:16:da', 'mtu': 1500, 'carrier': 0, 'ifname': 'v01', 'num_tx_queues': 1, 'index': 6, 'ifi_type': 1, 'ports': [], 'vlans': [], 'ipaddr': [], 'qdisc': 'noop', 'neighbours': [], 'txqlen': 1000, 'broadcast': 'ff:ff:ff:ff:ff:ff', 'operstate': 'DOWN', 'num_rx_queues': 1, 'flags': 4098, 'family': 0} >>> </code></pre> >>> with ipdb_namespace1.interfaces.vp01 as veth: ... veth.add_ip('192.168.1.37/24') ... veth.up() ... </code></pre> {'kind': 'veth', 'promiscuity': 0, 'group': 0, 'ipdb_scope': 'system', 'ifi_type': 1, 'linkmode': 0, 'ipdb_priority': 0, 'carrier_changes': 1, 'num_rx_queues': 1, 'address': '26:29:f4:06:8a:ea', 'mtu': 1500, 'carrier': 0, 'ifname': 'vp01', 'num_tx_queues': 1, 'index': 5, 'ports': [], 'vlans': [], 'ipaddr': [], 'qdisc': 'noop', 'neighbours': [], 'txqlen': 1000, 'broadcast': 'ff:ff:ff:ff:ff:ff', 'operstate': 'DOWN', 'flags': 4098, 'family': 0} >>> </code></pre> >>> with ipdb_namespace1.interfaces.lo as i: ... i.up() ... </code></pre> {'flags': 8, 'promiscuity': 0, 'group': 0, 'ipdb_scope': 'system', 'ifi_type': 772, 'linkmode': 0, 'ipdb_priority': 0, 'carrier_changes': 0, 'address': '00:00:00:00:00:00', 'mtu': 65536, 'carrier': 1, 'ifname': 'lo', 'num_tx_queues': 1, 'index': 1, 'ports': [], 'vlans': [], 'ipaddr': [], 'qdisc': 'noop', 'neighbours': [], 'txqlen': 0, 'broadcast': '00:00:00:00:00:00', 'operstate': 'DOWN', 'num_rx_queues': 1, 'family': 0} >>> </code></pre> At this point, we can open a shell and directly check the configuration for v01</code> interface: kintanu:~# ip addr show v01 </code></pre> 6: v01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether f2:2d:aa:9d:16:da brd ff:ff:ff:ff:ff:ff inet 192.168.1.36/24 scope global v01 valid_lft forever preferred_lft forever inet6 fe80::f02d:aaff:fe9d:16da/64 scope link valid_lft forever preferred_lft forever </code></pre> We can also, still from the shell, check the available interfaces inside namespace1</code>: kintanu:~# ip netns exec namespace1 ip addr show </code></pre> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 5: vp01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 26:29:f4:06:8a:ea brd ff:ff:ff:ff:ff:ff inet 192.168.1.37/24 scope global vpeer01 valid_lft forever preferred_lft forever inet6 fe80::2429:f4ff:fe06:8aea/64 scope link valid_lft forever preferred_lft forever kintanu:~# </code></pre> A system uses its routing table to determine which network interface to use when sending packets to remote systems. Let's make all traffic leaving namespace1</code> to go through v01</code>: </a> kintanu:~# ip netns exec namespace1 ip route add default via 192.168.1.36 </code></pre> I told you I often use LXC containers when I'm working. For that reason, I always have a bridge configured and up. Then, all the containers are connected to the internet, via the bridge, as ports</code>. Let me show you: kintanu:~# ip addr show br0 </code></pre> 4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 3c:97:0e:65:c0:27 brd ff:ff:ff:ff:ff:ff inet 192.168.1.34/24 brd 192.168.1.255 scope global br0 valid_lft forever preferred_lft forever inet6 fe80::3e97:eff:fe65:c027/64 scope link valid_lft forever preferred_lft forever </code></pre> kintanu:~# brctl show </code></pre> bridge name bridge id STP enabled interfaces br0 8000.3c970e65c027 no eth0 vethLMRO1N kintanu:~# </code></pre> The br0</code> interface is the interface I use for going outside. In fact, br0</code> is a bridge. eth0</code> and vethLMRON</code> are interfaces, ports</code>, attached to that bridge</a>. v01</code> should be attached to the br0</code> bridge, otherwise, v01</code> won't be able to send packets to the outside world: </a> kintanu:~# brctl addif br0 v01 kintanu:~# brctl show </code></pre> bridge name bridge id STP enabled interfaces br0 8000.3c970e65c027 no eth0 vethLMRO1N v01 kintanu:~# </code></pre> ![Veth pair device added to bridge]({filename}/images/ns.png) If everything went fine, you should be able to ping an external host from `namespace1`: ``` kintanu:~# ip netns exec namespace1 ping 8.8.8.8 ``` ``` PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. From 192.168.1.34: icmp_seq=1 Redirect Host(New nexthop: 192.168.1.1) 64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=104 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=105 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=105 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=104 ms ^C --- 8.8.8.8 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 104.190/104.859/105.375/0.595 ms kintanu:~# ``` We've just seen how to create a network namespace, and link that network namespace to the outside world. When playing with LXC containers, you will also heard about control groups</code>, a kernel mechanism that will allow you to allocate resources among user-defined groups of tasks (processes) running on a system. That will be the topic of the next chapter. All the diagrams were made using [Asciiflow](http://asciiflow.com/) Introduction 2015-07-24T00:00:00+00:00 Hello, I'm Patrick, born and raised in central Africa</a>, currently based in West Africa, Dakar Senegal</a>. I am an experienced software engineer, passionate about free software</a>. I have a strong background in developing backend solutions. With almost 10 years of hands-on experience in the field, I have successfully contributed to free software</a> and I have successfully developed and deployed numerous</a> software</a> projects</a> across</a> various</a> industries</a>. My expertise lies in free software</a>, Python programming</a>, databases</a>; enabling me to tackle complex challenges and deliver high-quality, scalable software solutions. I thrive in dynamic and collaborative environments, working closely with cross-functional teams to understand requirements, design architectures, and implement robust solutions. Throughout my career, I have demonstrated a proven track record of delivering projects on time and within budget, while ensuring adherence to best practices and industry standards. I am adept at analyzing existing systems, identifying areas for improvement, and implementing enhancements that optimize performance and user experience. Over the course of my professional journey, I have had the privilege of working as a programming instructor</a> for diverse groups of learners, ranging from beginners to experienced professionals. My teaching</a> approach is centered around creating an engaging and supportive learning environment</a>, where students can grasp complex concepts with ease and apply them effectively. Beyond technical skills, I am a team player, strong communicator and effective problem solver. I excel at sharing knowledge, at translating complex technical concepts into clear & non-technical language, fostering effective communication between stakeholders, clients, and development teams. I am constantly exploring emerging technologies and staying up-to-date</a> with industry trends, as I believe in continuous learning and professional growth. I am eager to leverage my skills and expertise to contribute to a dynamic organization, driving innovation and delivering exceptional software solutions.

Use cases for C pointers:</h2>
All modern programming languages have pointers in some form or other. Pointers are used everywhere there is a program, even when we don't see them. Let's explore some of those use cases.</p>

Interfacing with hardware:</h3>
C Pointers are often used in low-level programming to interact with hardware devices and memory-mapped registers. They provide a way to access specific memory addresses and manipulate hardware directly.</p>

Going further:</h2>
To learn more about C pointers, we recommend the following pointers</strong>:</p>
0x7ffc6155fd68</a></li>
0x7ffc6155fd69</a></li>
0x7ffc6155fd70</a></li>
0x7ffc6155fd71</a></li> </ul>

Types:</h2>
Variable, a named location in the computer's memory where data can be stored and accessed. Type, define the size, the format, the behavior, the operations that can be performed on the variable. Let's talk about the common types in C programming.</p>

Primitive types:</h2>
Primitive data types, or "basic data types" or "fundamental data types" are the most basic data types that can be used for representing simple values such as numbers, characters, etc... The primitive data types are...</p>

Composite types</h2>
Any data type which can be constructed using the language's primitive data types and other composite types.</p>

On a completely differnt note:</h1>

Des organismes aux algorithmes, l'odyssée computationnelle du vivant | Xavier Berthet | TEDxINPENSAT</a></li>
The first DjangoCon Africa, in Zanzibar, Tanzania.</a></li> </ul>

First solution:</h1>
Let's write a Python program to achieve this task. We'll use the following data set</a></p>

5.6. How good is our model:</h2>
We need to able to measure how good our model is. Among the methods available to achieve this task, let's use the most common and the most used: the Root mean squared error</a>.</p>

More on the topic:</strong></h3>
C syntax for functions returning function pointers</a></li> </ul>

3. Set a Tox environment and CI job:</h3>
Add Tox environment to automate the boring stuff and reduce the boilerplate</a>. And add a Gitlab CI job</a>. that will be in charge of running Mypy every time a new changeset is sent to the Gitlab server.</p>

Classes hierachy:</h2>
Within Pylint</a>, there is a tool named Pyreverse</a> that analyses Python code and extracts UML class diagrams and package dependencies. Using Pyreverse, we can have a clear view of all the classes and the relation between them:</p>
</p>

I want type annotations, what's next?</h2>
Mypy</a> is a static type checker for Python. Let's see what Mypy can to for us.</p>

Thanks:</h1>
Special thanks</a> to Orbifx, the Nature's emissary</a>, and Travis Kalanick</a>. They both gave me interesting ideas to help me finish this article. </p>

Do not forget:</h1> </center>

I hope you've learned something, and if you want to know more:</h3>
Processes</a></li>
Proc</a></li>
Proc man pages</a></li>
The "vsyscall" and "vDSO" segments</a></li>
GDB</a></li>
The jealous process</a></li> </ul>

Nskm

C Pointers (WIP)

C

Decorators

Test driven interview

Linear regression

Scafold

Yasnippet

Wall of Fame

Writing

Function pointers

My journey to type checking 7521 lines of Python

To type or not to type? That is the question

Elephant in the room (part 1)

Django single file project. Again

Free and Open Source Software

Threads

Django single file project

Make

How to create processes

Processes

Network namespaces

Introduction

Nskm

C Pointers (WIP)

Introduction:</h2> A pointer is a variable that contains the memory address of another variable. A pointer allows us to indirectly</em> access and manipulate the value of the variable that is pointed to. Like other variables, C pointers have a type.</p> </p>

C pointer operators:</h2> C offers operators for pointer manipulation:</p>

Use cases for C pointers:</h2> All modern programming languages have pointers in some form or other. Pointers are used everywhere there is a program, even when we don't see them. Let's explore some of those use cases.</p>

Manipulating array and string elements:</h3> C Pointers can be used to access and manipulate elements in arrays and strings. C Pointers provide a way to efficiently iterate over arrays and perform operations on each item of the sequence.</p>

Calling functions:</h3> A function pointer, is a pointer which point to the memory address of a function instead of pointing to the address of a data object. They are used to:</p>

Interfacing with hardware:</h3> C Pointers are often used in low-level programming to interact with hardware devices and memory-mapped registers. They provide a way to access specific memory addresses and manipulate hardware directly.</p>

Going further:</h2> To learn more about C pointers, we recommend the following pointers</strong>:</p>

C

The setup:</h2> To write C programs, we need:</p>

Types:</h2> Variable, a named location in the computer's memory where data can be stored and accessed. Type, define the size, the format, the behavior, the operations that can be performed on the variable. Let's talk about the common types in C programming.</p>

Primitive types:</h2> Primitive data types, or "basic data types" or "fundamental data types" are the most basic data types that can be used for representing simple values such as numbers, characters, etc... The primitive data types are...</p>

Void type:</h3> Void is a keyword to use as a placeholder where you would put a data type, to represent "no data". The void data type represents the absence of type. Void types are used in the following situations:</p>

Composite types</h2> Any data type which can be constructed using the language's primitive data types and other composite types.</p>

Constants:</h2> A fixed value that cannot be, that should not</em> be changed during the execution of a program. </p>

Decorators

1. A function:</h1> Let's suppose we want to join 2 paths:</p>

2. A decorator modifying the behaviour of the function:</h1> While it may seem confusing at first, a decorator is really just a function that takes a function and returns a function</em>. Let me explain. </p>

3. A decorator should keep important informations:</h1> Now we have another problem. Because we are literally replacing one function with another, we're also losing important informations like the docstring and the name: </p>

4. A decorator should receive a flexible number of argument:</h1> Now, let's suppose our original function can take many arguments:</p>

5. A decorator can also have parameters:</h1> Let's suppose people using the decorator want to be able to set what should the prefix used as the first parameter of our jp function. Let's suppose we want the decorator to receive one parameter, which is the prefix:</p>

9. More on this topic:</h1> I hope that you found this post helpful. To learn about this topic, please, browse the following links:</p>

Test driven interview

Let's commit what have been done so far.</h3> All the tests related to Product are passing. We can commit and push what we've done so far.</p>

Linear regression

5.1 Mean:</h2> The arithmetic</em> mean, is a measure of central tendency of a finite set of numbers: specifically, the sum of the values divided by the number of values:</p>

5.2 Variance:</h2> The variance is the measure of the spread</strong> in a data set. The variance is a measure of how far each number in the data set is from the mean, and thus from every other number in the data set. The higher the variance, the more spread out the variables are.</p>

5.3 Covariance:</h2> The covariance is a measure of the relationship between two variables. Covariance is the extent to which the variance in one variable depends on another variable. The higher the covariance, the stronger the relationship.</p>

5.4 Estimates:</h2> Now that we have a function for calculating the variance and another one for calculating the covariance, we can write a function that will calculate our slope and our intercept using this 2 formulas seen earlier:</p>

5.5. Prediction:</h2> Now that we have extracted our coefficients, let's see if we can make a prediction, we'll use the testing data set:</p>

5.6. How good is our model:</h2> We need to able to measure how good our model is. Among the methods available to achieve this task, let's use the most common and the most used: the Root mean squared error</a>.</p>

RMSE:</h3> One way (the most common way?) to assess how well our model fits our data set is to calculate the Root Mean Squared Error. The RMSE is:</p> the square root </li> of the mean </li>

Normalized RMSE:</h3> The NRMSE is the RMSE divided by Y_max - Y_min. The result will be between 0 and 1, where values closer to 0 represent better fitting models. The normalization facilitates the comparison between data sets with different scales:</p>

Scafold

Yasnippet

Wall of Fame

Writing

Links:</h1> To learn more on this topic, I strongly recommend the following links:</p>

Even more links:</h1> Things that were discussed during the talk:</p>

Function pointers

My journey to type checking 7521 lines of Python

Type hints addition:</h2> There are different options for adding type annotations.</p> 1. Manual</h3> We can do it manually: finding what are the types flowing through our program, I've been there, it's painful</em>. Literally:</p>

1. Manual</h3> We can do it manually: finding what are the types flowing through our program, I've been there, it's painful</em>. Literally:</p>

To type or not to type? That is the question

I need to know the types:</h2> When you read/debug/maintain a large software system you've not written, you'll often be in the position in which you ask yourself: "Those function arguments, those variables, what are their types?", or "Is this function returning something?".</p>

Use cases for C pointers:</h2>
All modern programming languages have pointers in some form or other. Pointers are used everywhere there is a program, even when we don't see them. Let's explore some of those use cases.</p>

Interfacing with hardware:</h3>
C Pointers are often used in low-level programming to interact with hardware devices and memory-mapped registers. They provide a way to access specific memory addresses and manipulate hardware directly.</p>

Going further:</h2>
To learn more about C pointers, we recommend the following pointers</strong>:</p>
0x7ffc6155fd68</a></li>
0x7ffc6155fd69</a></li>
0x7ffc6155fd70</a></li>
0x7ffc6155fd71</a></li> </ul>

Types:</h2>
Variable, a named location in the computer's memory where data can be stored and accessed. Type, define the size, the format, the behavior, the operations that can be performed on the variable. Let's talk about the common types in C programming.</p>

Primitive types:</h2>
Primitive data types, or "basic data types" or "fundamental data types" are the most basic data types that can be used for representing simple values such as numbers, characters, etc... The primitive data types are...</p>

Composite types</h2>
Any data type which can be constructed using the language's primitive data types and other composite types.</p>

5.6. How good is our model:</h2>
We need to able to measure how good our model is. Among the methods available to achieve this task, let's use the most common and the most used: the Root mean squared error</a>.</p>