Nieuws over AI en datascience

  • How to Create Beautiful Waffle Charts for Data Visualisation in Python
    by Andy McDonald on 9 februari 2023 at 06:13

    A Great Alternative to Pie Charts for Data VisualisationContinue reading on Towards Data Science »

  • We Should Have Seen ChatGPT Coming
    by Samantha Hodder on 9 februari 2023 at 02:07

    We’ve all been contributing to it for yearsContinue reading on Towards Data Science »

  • Creating Time Series Plots in Grafana
    by Wei-Meng Lee on 9 februari 2023 at 02:02

    Learn how to plot dynamic time series plots using Python and GrafanaContinue reading on Towards Data Science »

  • 4 Faster Pandas Alternatives for Data Analysis
    by Chengzhi Zhao on 9 februari 2023 at 02:00

    Performance Benchmark on Popular Data Analysis ToolsContinue reading on Towards Data Science »

  • A Step-by-Step Guide to Develop a Map-Based Application (Part II)
    by Jacky Kaub on 9 februari 2023 at 01:46

    How to use React to build web appsPhoto by Lautaro Andreani on UnsplashMaps are a powerful tool for visualizing and understanding geographic data but they need specific skills to be designed efficiently.In this step-by-step guide, we are going to take a deep dive into building a map-based application to show the customer's prices of gas stations around them. We will cover the different key steps of a product, from original proof of concept (POC) to the minimum viable product (MVP)Articles in the series:Part I: The proof-of-concept — Build a minimalist demoPart II: How to use React to build web apps (Static Layout)A bit of context around this articleIn this article, we will continue our work on the Gas Station Finder Application that was initiated previously. This time, we will focus on preparing the first version of the web application that can run locally using static datasets. The web application will be built using the React framework and this article will provide as much detail as possible for someone who is new to the framework.I decided to go with React for this application as it is a compelling framework that only a few data scientists/data analysts are currently using. If you have been ever frustrated by the rigidity of other frameworks, I hope you will find in this article an alternative way to design your web application. You will see that the learning curve might look a bit steep at the beginning, (especially if you are not familiar with HTML/CSS/javascript), but once mastered, having this framework in your toolbox will open up a lot of opportunities for building dynamic, user-friendly and scalable web applications.Note: The article will not include all the code, only snippets to illustrate key principles, but the full code can be found on my GitHub page.By the end of the article, we will have a layout of our React Application, and in the next part, we will focus on adding interactivity to it.The application “static” render by the end of the article, Author IllustrationWhy use React?There are many tools and frameworks available to build nice and powerful web applications. Some of the most adopted frameworks in the data science ecosystem include Dash or Streamlit. One of the big advantages of those tools is their simplicity and ease of use, which can be very efficient and sufficient for some of your use cases, especially if you are using mainly python.Nevertheless, it comes with a trade-off on flexibility and customization, and if your application starts to be more complex, the amount of time to develop your features within those frameworks will grow exponentially.React, on the other hand, is a framework that is more performant and allows for more flexibility. It also encourages the use of a modular and component-based architecture, which is appreciable when having to add new features and do the maintenance of the project. Finally, it is part of the JavaScript ecosystem, which is one of the largest and most popular ecosystems in the world.Despite its advantages, you might be reluctant in learning the framework, especially if you come from the python world. Not only would you have to learn the framework itself, but also probably some javascript, HTML, and CSS, which can be very time-consuming.Regarding HTML and CSS, a basic knowledge of those languages is in any case needed even if you try to go deeper into your application (even in dash or streamlit). Javascript might be a bit more challenging, especially if you are not so much familiar with object-oriented and asynchronous programming, but once learned, it will potentially give you access to a more easy and efficient way of developing web apps (as well as sharpening your programming skills)Setup the ProjectTo start a project powered by react, you first need to install Node.js on your computer, which is the runtime environment used to run javascript.Node.js can be downloaded here.Side note: Things are changing fast. I wrote this article in March 2023 and I am using React 18.2.0. If you read this article years after its publication, it might be a bit outdated. So be careful with what you will read below.Create a new projectOnce installed, we will use npx/npm (roughly the equivalent of pip in python) to set up a react project via the command “create-react-app”npx create-react-app fuel-station-frontRunning this command in a CLI will automatically download all the required packages and set up your project in the following folders:fuel-station-front/ |-- node_modules/ |-- public/ |-- index.html |-- favicon.ico |-- ... |-- src/ |-- App.js |-- index.js |-- ... |-- package.json |-- package-lock.json |-- README.mdThis might look complicated, let us go into details file by file:node_modules/: This folder contains all the packages and dependencies that you will add to your project using npm. In general, you should leave this folder untouched.public/: This folder contains static files used for your project. It is populated by default with basic files to run the default page that is generated when you initiate a new project.src/: This folder is the core of your application and contains its source code. This is where we will write the code of our application. It is populated initially with files to run the default page that is generated when you initiate a new project.package-lock.json: This file is the equivalent of requirement.txt. It contains all the dependencies and exact versions of the packages that are installed using npm.package.json: This file is the equivalent of setup.py and it specifies project metadata.Running the projectIn this part of the life cycle of the project (basically building and testing locally), we will use a development server which is a built-in feature of the create-react-app tool. This is good for making quick developments but is not robust enough for production purposes, and we will talk about this in another article.To run your test server, simply go to your project folder and run the following command:npm startThis will automatically open a new webpage in your default browser with the default page imitated in any new react project.the “localhost” means we are running the server on our own computer, while “:3000" indicates the port number of the application.The react default app, , Author IllustrationInstalling required packageReact has a large ecosystem and many packages are open sources and will simplify your life.In particular, for this project, we are going to use react-plotly.js, a wrapper of plotly made for react applications.npm install react-plotly.js plotly.jsPart of very interesting packages, I can also mention react-router-dom, a powerful URL mapper to manage multi-page applications, and uuid, a package designed to generate automatic ids based on the latest standards. I will not use them today, but this is part of the must-known packages.Improve the coding experienceI will code the project using VSCode. Many extensions are available in order to simplify your life while developing a React web app.Let’s talk briefly about some of them.ES7 React/Redux/GraphQL/React-Native snippets: This extension provides a lot of shortcuts to quickly set some template code.ES7 in action, Author IllustrationEmmet: This plugin allows for fast coding by providing also a lot of shortcuts and abbreviations to automatically generate code.Emmet in action, Author IllustrationPrettier: This extension automatically formats your code when you save your files.Build the layoutNow that our environment is properly set up, we can start to build our application. As mentioned in the previous part, this will be done mainly after modifying the files in the src/ folder.Let’s look a bit at what is inside there:|-- src/ |-- App.js |-- index.js |-- App.test.js |-- reportWebVitals.js |-- setupTests.js |-- index.css |-- App.css |-- logo.svgThe only file of interest for us will be App.js and we will go into more detail about it later. Regarding the other files, quickly:index.js is responsible for the rendering of the application. We will not touch this filereportWebVitals.js is used to report the performance metrics of the web page to an analytic service, and in our level of use of React, will not be used.setupTests.js is a file used to set up a testing environment. Also out of the scope of this article and we will leave it untouchedlogo.svg is simply an image used in the default page when you create the project and can be deletedApp.css and index.css contain some basic .css for your web app, that could eventually be modified to give another style if you need to.Application ArchitectureIn react, we build an application by assembling custom components. As a best practice, each component will be made of two files:a .js file used to code the logic of the component: what it does, how it manages its internal state, and how it propagates its state to its parents and children componentsa .css file used for the esthetic and formatting of the componentLet’s take the example of our application and see how our logic will apply. We want our App to have basically 3 main components:A search area form will be used to look for a place and a type of fuel.A map that will display the stations, the same way we built it in the previous articleA table to summarize quickly the main information about the stations around.This will look something like this:|-- src/ |-- App.js |-- App.css |-- AppComponents/ |-- StationsFilter/ |-- StationsFilter.js |-- StationsFilter.css |-- CustomTextBoxes/ |-- ... |-- StationsMap/ |-- StationsMap.js |-- StationsMap.css |-- StationsTable/ |-- ... |-- index.js |-- ...This might look like a lot of files and folders, but it will help keep your project clean and tidy, which will make your life much easier in the future if you need to go back to it to modify or improve your app.Application basic layoutAt this point, you might want to build a rough layout of what your app could look like. If you don’t have a web designer to help, the best is to look at examples of modern apps and get inspiration from them. The Internet is full of templates to build nice components, and with basic .css knowledge and the help of those templates, you will already be able to do nice and modern elements.As we start the project, we want to build a primary basic view just to have a rough idea about how to organize our components. Half of the job will be basically done with such a basic layout as a web application is basically made of boxes in boxes in boxes…For our web app, let’s keep things simple:Basic Application layout, Author IllustrationBefore going further, we also need to talk about design paradigms. In HTML/CSS, you have many ways of organizing your components. My favorite paradigm is the Flexbox paradigm, which provides an easy way to organize boxes within a container box.Using this paradigm, you set the organization style of your “container box”:How are aligned the children's boxes, in rows, or in columns?Within this structure, how are organized the children's boxes on the main axis?Within this structure, how are organized the children's boxes in the orthogonal axis?The layout used above follows the Flexbox paradigm.The gray area is the main container. We want it to contain two “div” (which are the box in HTML language) organized in columns: One containing the header (a title with sometimes some menu options) and one containing the main components.That div will be organized in a row in such a way that it divides the space in two: in the left part, we will have a div (in orange) for the filtering component and the table with the prices, and in the right, the map component which will take the remaining space.export default function App() { return ( // the gray box, we want its "organisation" to be in column <div className="App"> //the header, being the first blue box <header className="header"> <h1 className="h1-searchbar">Gas Station Finder</h1> </header> //the second blue box, containing the main components, which will be organised in row <div className="main-components"> //Inside that blue box, we have another box "left-section" (the orange box) containing the Filter and the Table <div className="left-section"> <StationsFilter /> <StationsTable /> </div> //Finally we also have the other component, our map, inside the second blue box <StationsMap /> </div> </div> );}<StationsMap />, <StationsTable />, and <StationsFilter /> will be the custom components that we are going to build in the coming section.App.css will contain the .css of our container boxes, let’s have a look!.main-components{ height:90vh; width:100%; display: flex; flex-direction: row; align-items: center;}.left-section{ height:100%; width: 39%; background-color: #f4f4f4; display: flex; flex-direction: column; align-items: center;}display: flex indicates that the containers are “flex-boxes” and that the children's boxes will be organized following this paradigmflex-direction indicates if the boxes are organized in a row or in a columnalign-items organize the alignment in the secondary directionjustify-content (not used here) organize the alignment in the main directionInitiate the componentsNow that all of the above is set up, let’s initialize our components with a basic version for all of them, which we will improve over time.In StationsFilter.js, we are going to use the shortcut “rfc” (as shown in the .gif in the previous section) that will directly generate the template for our component, and also add an import statement for the corresponding .css file.import "./StationsFilter.css";export default function StationsFilter() { return ( <div>StationsFilter</div> );}This little piece of code indicates with “export default” that the function StationsFilter can be later directly imported in another file like:import StationsFilter from 'pathtofile/StationsFilter';and used in its own HTML tag:<StationsFilter />Let’s make the same template for StationsMap.js and StationsTable.js, I will pass on the code snapshot here, which is exactly identical to what we did before.Working on the Filter LayoutWe want the filter to be made of two components:One to look for a place by postal codeOne to select the fuel typeimport "./StationsFilter.css";export default function StationsFilter() { const fuelTypes = ["E10","E85","Gazole","GPLc","SP95","SP98"] return ( <form className="search-form"> <input type="text" placeholder="Postal Code" className="general-input" /> <select value="SP98" className="general-input"> {fuelTypes.map(e => <option value={e}>{e}</option>)} </select> <button className="send-request-button">Find Stations</button> </form> );}Let's have a look in detail:We first define a box of type “form” in which we will put our form fields and the buttonWe create an input of type “text” used to receive text. We place a placeholder name “Postal Code” which will appear to give an indication to the customers and a className for the .cssWe create a Dropdown with the tag <select>.Inside this Dropdown component, we need to add the options as “children” tags. This is done by iterating via an array (the equivalent of a python list) to automatically generate all the possibilities.const fuelTypes = ["E10","E85","Gazole","GPLc","SP95","SP98"]<select value="SP98" className="general-input"> {fuelTypes.map(e => <option value={e}>{e}</option>)}</select>is equivalent to:const fuelTypes = ["E10","E85","Gazole","GPLc","SP95","SP98"]<select value="SP98" className="general-input"> <option value="E10">E10</option> <option value="E85">E85</option> <option value="Gazole">Gazole</option> <option value="GPLc">GPLc</option> <option value="SP95">SP95</option> <option value="SP98">SP98</option></select>We use the “map” version because it is more compact, and potentially easier to maintain: if a new fuel type appears, we can simply modify our fuelTypes array and it will generate automatically the corresponding options.Finally, we create a button that will be later the entry point for sending a request to the server and retrieve some data to displayNote: currently nothing is interactive, this will be detailed in the next article.Note 2: I will not detail the .css here, for most of it I took it by looking for inspiration on the internet and it is not necessarily very interesting to detail here.This is how looks the component on its own:The Filter Component, Author IllustrationBuilding the MapIn order to build the map, we are going to use the react-plotly plugin.In react, plotly figures are basically made the same way as in python, with traces and a layout. The main difference is that instead of using a wrapper to build our charts (like go.Scatter) we need to provide the data in the JSON format (a dictionary with keys/values).If you are used to building your charts in python, don’t panic, it is actually very simple to convert your python figure to a react figure.As a starting point for now and for the illustration, I am going to show you how to copy our python figure to a react one. Later, when we will add interactions, we will rebuild this map adding good data feeds. This is the map we want to copy from our python code:The map widget from the python POC, Author IllustrationFirst step: let’s create a .json copy of our chart:fig_json_path = "fig.json"#Convert the file to a json string, and saved in in the computerfig_as_json = fig.to_json()with open(fig_json_path, 'w') as f: f.write(fig_as_json)If you want to have a look at how looks your figure as a python dictionary, you can also do fig.to_dict(). In the case of the figure above, this will give something looking like this:{ "data":[trace1, trace2 ...], "layout":figure_layout}with, for example, taking the layout and one of the traces of our figure:trace_stations = { 'lat': stations_lat, 'lon': stations_lon, 'marker': {'color': stations_price, 'colorscale': colorscale 'size': 40}, 'mode': 'markers', 'opacity': 0.7, 'showlegend': False, 'text': stations_price, 'type': 'scattermapbox', 'uid': '911aadec-c42e-4eb7-968a-6e489606224e'}figure_layout = { 'height': 600, 'width': 600 'mapbox': {'accesstoken': token, 'center': {'lon': center_lon, 'lat': center_lat}, 'style': 'streets', 'zoom': zoom, 'bearing': 0, 'pitch': 0},}Most of the fields are generally familiar if you are used to plotly. Note the field “type” which specifies the type of trace you are building. In our case “type”:”scattermapbox”, equivalent to what we did in python with (go.Scattermapbox() )This can be used directly in react and it is reproducible for any type of figure making it very easy to go from a proof of concept in python to its counterpart in react.Let’s copy the full JSON we generated before in our project and include it in the folder of the component StationsMap.|-- src/ |-- App.js |-- App.css |-- AppComponents/ |-- StationsHeader/ |-- StationsMap/ |-- StationsMap.js |-- StationsMap.css |-- fig.jsonWe can now simply import the file in our StationsMap.js script and use it in a react-plotly Figure.import "./StationsMap.css"import Plot from "react-plotly.js";import figJson from "./fig.json"export default function StationsMap() { const data = figJson["data"] const layout = figJson["layout"] layout["autosize"]=true layout["margin"]={"b":0,"t":0,"l":0,"r":0} delete layout.template delete layout.width delete layout.height return ( <Plot data={data} layout={layout} style={{ height: "100%", width: "100%" }} useResizeHandler={true} /> )}Let’s have a look in detail:First, we import simply our JSON object and store it in the variable figJson. Done this way, the figJson is a javascript Object, which is the equivalent of Python dictionaries which will make the job for our current static view.import figJson from "./fig.json"Then, as we could do in python, we update our layout Object with extra information and remove some keys/values. In my case, I set the plotly layout to “autosize” which means it will take all the space available, remove all the margin around the figure and delete the key/values for “template”, “width”, and “height ” that where generated by default from the python figure. layout["autosize"]=true layout["margin"]={"b":0,"t":0,"l":0,"r":0} delete layout.template delete layout.width delete layout.heightFinally, I declare that the component returned by my StationsMap() function is a plotly component including all the traces from my son and the modified layout.return ( <Plot data={data} layout={layout} style={{ height: "100%", width: "100%" }} useResizeHandler={true} /> )Note that you can pass parameters to a component directly inside the tag using the format:<MyComponent param1={param1} param2={param2} />In the case of a plotly component, we pass a parameter data which is a list of objects representing the different traces of the chart, a parameter layout which is an object containing the layout information of the chart, and extra parameters like “style” or “useResizeHandler” which are parameters to indicate that the div generated by plotly should take all the remaining space.How plotly is rendered as a component of the app, same as the python version, Author IllustrationBuilding the TableWe are missing one last component to complete our App Layout: a table summarizing the information of the surrounding stations.Tables are components that can be difficult to design because we need to find the right level of information to show. They are usually more difficult to interpret than charts, but they are still very useful when you want an overall view of the details of your data.In the case of our component, we want to provide some of the raw data available such as:the address of the stationthe price of the fuel of interest which is expressed in €/L.On top of that, we can as well transform the data to be more concrete for a user, for example:the total price of an average full tank (say 60L)he potential saving or loss in comparison to the average places of the surroundingand why not add a Google Map link so that they can be redirected to Google services to get more information about the provided address?This looks like nothing, but putting yourself in the shoes of your customers is very important. The more clarity you bring to them in one look at the data, the more they will like your application and come back.Given that we are still in a “static view mode”, I will generate a sample of data in python based on the default map and use that at first for my layout. There are multiple ways of representing a table in a json, I chose to use a list of Objects that I find personally more convenient:dataTable = [ { "address":address1, "price_per_L":price_per_L1, "price_tank":price_tank1, "delta_average":delta_average1,"google_map_link":google_map_link }, { "address":address2, "price_per_L":price_per_L2, "price_tank":price_tank2, "delta_average":delta_average2, "google_map_link":google_map_link }, etc...]I’ll add some tips:If you are more comfortable in Python than in Javascript, it is fine. In that case, make all your data transformations in Python. We will see in one of the coming articles how to connect our React application to a Python API where most of the data transformation logic will be handled.Using pandas, you can very easily convert a DataFrame to a list of objects and dump it as a json using:with open("sample_table.json", "w") as f: json.dump(df.to_dict("records"), f)An example of DataFrame in python, before dumping in jsonAs with the plotly figure, let’s copy the file to the relevant folder|-- src/ |-- App.js |-- App.css |-- AppComponents/ |-- StationsHeader/ |-- StationsMap/ |-- StationsTable/ |-- StationsTable.js |-- StationsTable.css |-- sample_table.jsonNow that we have a base of work, let’s code the table. This is the general layout in HTML:<table className="price-table"> <thead> <tr> <th>Address</th> <th>Price</th> <th>Full Tank</th> <th>Saved</th> </tr> </thead> <tbody> <tr> <td <td>Some values</td> <td>Some values</td> <td>Some values</td> <td>Some values</td> </tr> <tr>...</tr> <tr>...</tr> </tbody></table>The <table> tag is used to initialize the table. We then define the table line by line. We start with the header which is a special part, indicated by the tag <thead>, and later by the body (the different rows of the table) in the <tbody> tag.Then each line is defined in the same way. <tr> is used to define a full row, and <th> a cell in that line. This means that if you have 4 cells in your header, you should define your cells in your rows with also 4 cells.Now let’s use our static data. As in the last part of the graph, we start by importing the pre-generated .json with the data:import tableJson from "./sample_table.json"Remember that this tableJson is an Array of Objects. We can use again the Array.map() method to iterate through the Objects of the array and generate dynamically our table.<tbody> {tableJson.map((row) => { return ( <tr key={row.address}> <td>{row.address}</td> <td>{row.price_per_L.toString()+" €/L"}</td> <td>{row.price_tank.toString()+" €"}</td> <td>{row.delta_average+" €" }</td> </tr> )})}</tbody>A few comments here:As we generate dynamically the rows of the table, React will generate a warning if we are not identifying each row with a unique key. I decided to pass the address which is in my case unique, but another way would have been to generate them automatically using an external lib like uuidNote that in JSX logic, the variables inside the tag have to be expressed using {} to indicate that they are variables (and their value should be accessed in the tag or in the parameters of the tag)We access the element of an Object in javascript the same way we do in python, so you should be familiar with the notation if you come from the python fieldI’ll never repeat it enough but keep a clean format. In this case, I don’t display only the raw numbers, but I add a suffix (€ or €/L) which makes the data more comprehensiveWe have now a very simple version of the table which looks like this, after a bit of formatting with some .css:Our data table, after some formatting, Author IllustrationThis version is not a bad start, but we can certainly improve it. First, we can add the links to Google Maps as mentioned above. You can pre-compute them from the address of the station using the below formula:google_link = f'https://www.google.com/maps/search/?api=1&query={address.replace(" ","+")}'Then we can generate a hyperlink from the cell of the address in our table passing that google link in a <a href> tag: <td> <a href={row.google_map_link}>{row.address}</a> </td>We want to make a last improvement to this table. You might have noticed I added in my python table a column named “better than average”? This is a value that will help us create a color mapping to quickly see the cheap stations to the expensive ones. Let’s create a variable color in our mapping function that is by default black, but that can become red or green depending on the “better than average” field, and let's pass that color to the style of the last cell of each row: if (row.better_average == 1) { color = "green"; } else if (row.better_average == -1) { color = "red"; } ... <td style={{ color: color }}>{row.delta_average + " €"}</td>And let’s have a look at our final result:Our final Table component, Author IllustrationOur table is ready, we can now see very easily which stations are the more competitive in terms of price, and by a simple click, check their location in Google Maps!Final App LayoutAt that point, our layout is finalized. We already implemented the main App layout in the first chapter:export default function App() { return ( // the gray box, we want its "organisation" to be in column <div className="App"> //the header, being the first blue box <header className="header"> <h1 className="h1-searchbar">Gas Station Finder</h1> </header> //the second blue box, containing the main components, which will be organised in row <div className="main-components"> //Inside that blue box, we have another box "left-section" (the orange box) containing the Filter and the Table <div className="left-section"> <StationsFilter /> <StationsTable /> </div> //Finally we also have the other component, our map, inside the second blue box <StationsMap /> </div> </div> );}So we can simply look at http://localhost:3000/ and check what it looks like:The final app layout, Author IllustrationNote that if you are not satisfied with this layout, given the modularity of our code, it is very simple to change it to a different configuration. (In our next article, we will also talk about application responsiveness)ConclusionIn this article, we continued the work initiated last time about our Station Finder Application. In particular, we started to look in detail at the framework React to build robust and highly customizable Web Applications.We reach a point where we can run a static version of the application on a test server locally, but we are still far from having an application fully operational.Our React prototype is not done yet, and in the next article, we will continue to explore React, focusing this time on interactive features.A Step-by-Step Guide to Develop a Map-Based Application (Part II) was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

  • 4 Essential Techniques You Must Learn as a Python Beginner
    by Murtaza Ali on 9 februari 2023 at 01:42

    And how to use them so you can ace that next job interview with easeContinue reading on Towards Data Science »

  • Learn Discrete Fourier Transform (DFT)
    by Omar Alkousa on 8 februari 2023 at 13:38

    Mathematical and Coding PerspectiveDigital signal processing (DSP) is the computation of mathematical methods used to manipulate signal data [1]. One of the most important tools in digital signal processing is the Discrete Fourier Transform (DFT). It is usually used to produce a signal’s frequency-domain (spectral) representation [2].In this post, we will discuss how DFT works and how to implement it to output the spectrum of the signals.Photo by Pawel Czerwinski on UnsplashDiscrete Fourier Transform (DFT)The Fourier Transform is the mathematical backbone of the DFT and the main idea behind Spectral Decomposition which concludes that a signal is nothing but a sum of sinusoids of different frequency components [3]. Since all of the signal data we are working with are in digital form, a signal is a set of samples in the time domain. The Fourier transform on such discrete signals can be done using DFT, which can be used to switch back and forth between the time and the frequency domains. The time domain contains the samples of the signal, whereas the frequency domain represents the spectrum of the sinusoids that construct the signal [4]. The image below describes the relationship between time and frequency domains using DFT and IDFT.The relationship between time and frequency domains [4]. [Image by the Author]Mathematically speaking, if we have a signal (xn) with N samples, the DFT of this signal is defined as [5]:The DFT equation [5]Where:N: Number of samplesn: Current samplek: Current frequency where k ∈ [0, N−1]xn: The sine value at sample nXk: The DFT which includes information on both amplitude and phaseThe output of the DFT (Xk) is an array of complex numbers that hold the information on the frequencies, amplitudes, and phases of sinusoids that construct the input signal. The first half of the DFT array (Xk) contains the positive frequency terms, while the second half contains the negative frequency terms. Also, when the input signal is only a real-valued signal, the first half is the conjugate of the second half of frequency terms and the spectrum is symmetric. So, we focus only on the first half (the positive frequency terms) in the case of real-valued signals [5]. The figure below represents each of the positive and the negative frequency terms in case the number of the input samples (N) is odd or even.Positive and negative frequency terms [5]. [Image by the Author]The amplitude and the phase of each sinusoid that adds up to construct the signal can be calculated from the complex array (Xk) as follows (Im, and Re, are for Imagery and Real parts of a complex number, respectively) [5]:The amplitude and the phase information can be calculated from these equations [5]Let’s Code:We will build a function that calculates DFT based on the first equation above. But first, we should generate the signal that we want to use as an input for DFT. We will generate a signal that is a sum of 3 sine waves with frequencies (1,20,10)Hz and amplitudes (3,1,0.5). The sampling rate will be 200 samples per second. For generating the signal, I used a class, Signal, that you can use as well following this GitHub gist. You can easily generate any signal but I used this Signal class for more controllability.Note:The DFT and IDFT functions we will write below are released under the MIT license and all credits go to the authors of this book.Both of the functions we will discuss are just to understand what is the mathematics behind DFT and what is the output of such transform. In practical use, there are faster and more efficient algorithms that can calculate the Fourier Transform, Fast Fourier Transform (FFT), and its inverse.Let’s get started…# Import the required packagesimport numpy as npimport matplotlib.pyplot as plt# Generate the three signals using Signal class and its method sine()signal_1hz = Signal(amplitude=3, frequency=1, sampling_rate=200, duration=2)sine_1hz = signal_1hz.sine()signal_20hz = Signal(amplitude=1, frequency=20, sampling_rate=200, duration=2)sine_20hz = signal_20hz.sine()signal_10hz = Signal(amplitude=0.5, frequency=10, sampling_rate=200, duration=2)sine_10hz = signal_10hz.sine()# Sum the three signals to output the signal we want to analyzesignal = sine_1hz + sine_20hz + sine_10hz# Plot the signalplt.plot(signal_1hz.time_axis, signal, 'b')plt.xlabel('Time [sec]')plt.ylabel('Amplitude')plt.title('Sum of three signals')plt.show()The signal we generated using our class Signal. [Image by the Author]Now we will build the DFT function to give us the sinusoids that construct the signal we generated above. Make sure to carefully read the comments on the code below as it helps you conduct the output of each line.# Build a function that calculates the discrete Fourier transformdef DFT(signal): # Number of samples, 100 samples in our example N = len(signal) # The samples from 0 to N-1, [0, 1, 2, ..., 199] in our example n = np.arange(N) # Generate the frequencies, [[0], [1], [2], ..., [199]] in our example k = n.reshape((N,1)) # e is a matrix of complex numbers with a shape of (N, N), (200, 200) in our example e = np.exp(-2j * np.pi * k * n / N) # dft is a matrix of complex numbers with a shape of (N,), (200,) in our example dft = np.dot(e,signal) return dft# Let's use the functiondft = DFT(signal= signal)# Calculate the amplitude spectrum of the signalamp = np.abs(dft)# Generate the frequency axisN = len(dft)n = np.arange(N)T = N/signal_1hz.sampling_ratefreq = n/T# Plot the spectrumplt.figure(figsize = (8, 6))plt.stem(freq, amp, 'b', markerfmt='o', basefmt='b')plt.xlabel('Frequency [Hz]')plt.ylabel('DFT Amplitude |X(freq)|')plt.title('Spectrum of the signal')The output of the DFT function is the spectrum of the signal. [Image by the Author]The x-axis contains the frequency components that construct the signal. The y-axis represents the strength of each frequency component. The lowest frequency component in the spectrum is usually called the Fundamental Frequency and the frequency component with the largest amplitude is called the Dominant Frequency [3]. In our example above, the 1Hz frequency component is the fundamental and the dominant frequency.We can notice the symmetry of the spectrum at half of the sampling rate (try different rates); this is usually called the Folding Frequency. When recording a real-world signal (f(t)) with FN as the highest frequency component, the folding frequency should never go below FN to retrieve all of the signal’s information. And that’s according to Nyquist-Shannon Theorem [5].We can get the actual amplitude of the spectrum by normalizing it to the number of input samples (N). But when we focus only on the first half of the spectrum, in case the input is a real-valued signal, we normalize the spectrum by N/2 [5]. The code below is to normalize the first half of the spectrum [0, N/2] and to plot the spectrum.# Get the length of one side of frequenciesn_oneside = N//2# Get the one side frequencyf_oneside = freq[:n_oneside]# Normalize the amplitude by N/2one_side_dft = dft[:n_oneside]/n_oneside# Plot the first halfplt.stem(f_oneside, np.abs(one_side_dft))plt.xlabel('Freq (Hz)')plt.ylabel('DFT Amplitude |X(freq)|')plt.title('The spectrum of the signal after normalization')plt.show()The spectrum of the signal after the normalization. [Image by the Author]Inverse Discrete Fourier Transform (IDFT)Similarly, as you can go from the time domain to the frequency domain, you can get back from the frequency domain to the time domain using the Inverse Discrete Fourier Transform. This process is very useful in signal processing when you want to filter specific frequency components from the signal using DFT and then retrieve the signal back to its time domain using IDFT. The IDFT can be calculated from the Xk sequence following the equation [5]:IDFT equation [5]Let’s build a function that can calculate the IDFT using the equation above.def IDFT(dft): # Number of frequencies, 200 components in our example N = len(dft) # The frequencies from 0 to N-1, [0, 1, 2, ..., 199] in our example k = np.arange(N) # Generate the samples, [[0], [1], [2], ..., [199]] in our example n = k.reshape((N,1)) # If your input was a first half spectrum, 2j should be 1j to retrieve the signal e = np.exp(2j * np.pi * k * n / N) # dft is a matrix of complex numbers with a shape of (N,), (200,) in our example signal = np.dot(e,dft)/N return signal# Apply the Inverse Fourier Transform on the spectrum [dft]sig = IDFT(dft)# Generate the time axis from sampling rate and length of dftN = len(dft)duration = N/signal_1hz.sampling_ratetime_axis = np.arange(0, 2, 1/200)# Plot the results of IDFT along with the original signalplt.plot(time_axis, sig,'b')plt.plot(time_axis, signal, 'r')plt.xlabel('Time [sec]')plt.ylabel('Amplitude')plt.title('Output of the IDFT')plt.show()The signal retrieved by the IDFT function and the original signal are the same. [Image by the Author]The limit of DFTIn the case of a signal with a big number of samples, the execution time of the DFT function will be long to apply the Fourier Transform on all of the data points of the signal. Fortunately, an efficient algorithm to calculate the DFT of the signal has been developed, the Fast Fourier Transform (FFT). This algorithm reduces the execution complexity from O(N²) to only O(NlogN), where N is the size of the data. The significant reduction of computation complexity by using FFT allows the wide use of Fourier Transform in engineering, science, and mathematics fields [5].Python provides multiple functionalities that the user can use to apply Fourier Transform using Numpy or Scipy python packages. The code below represents the comparison of time execution using the DFT function we built above, the FFT using the Numpy package [6], and the FFT Scipy package [7]. Using FFT from the Scipy package was the fastest.# Import the scipy packagefrom scipy.fftpack import fft# Estimate the execution time of DFT using the function we've builtprint('Execution time of DFT Function:')%timeit DFT(signal)# Estimate the execution time of DFT using FFT from numpy packageprint('\nExecution time of FFT using Numpy pacakge:')%timeit np.fft.fft(signal)# Estimate the execution time of DFT using FFT from scipy packageprint('\nExecution time of FFT using Scipy package:')%timeit scipy.fftpack.fft(signal)Execution time of DFT Function:17.3 ms ± 2.65 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)Execution time of FFT using Numpy pacakge:8.72 µs ± 2.2 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)Execution time of FFT using Scipy package:8.27 µs ± 137 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)ConclusionWe’ve learned how useful the Fourier Transform is in the signal processing field. And we’ve understood the main idea behind it.We’ve pointed out the mathematical perspective of the Discrete Fourier Transform and how it can be calculated for a discrete signal.We’ve built a function that calculates the DFT using the mathematical equation and we applied the function to a signal we generated using a class Signal we’ve built in this previous post.We’ve learned that the output of the DFT is an array of complex numbers with N number of elements, the same as the number of samples of the input signal. These complex numbers hold information on the amplitude and the phase of the spectrum.We’ve seen that the output of the DFT will be symmetrical to half of the sampling rate if the input signal is a real-value signal. And this is why we focus only on the positive frequency components.An equation to calculate the Inverse Discrete Fourier Transform, IDFT, has been pointed out. And we’ve built a function to calculate the IDFT to retrieve the original signal from the spectrum.We’ve discussed the limitation of the functions we’ve built and there is an efficient algorithm to calculate the Fourier transforms, Fast Fourier Transform.References[1] R. Toulson, R., & Wilmshurst, T. (2012). An Introduction to Digital Signal Processing. Fast and Effective Embedded Systems Design (pp. 219–242). Elsevier. https://doi.org/10.1016/B978-0-08-097768-3.00011-8[2] T. Giannakopoulos, T., & Pikrakis, A. (2014). Signal Transforms and Filtering Essentials. Introduction to Audio Analysis (pp. 33–57). Elsevier. https://doi.org/10.1016/B978-0-08-099388-1.00003-0[3] Downey, A. (2016). Sounds and Signals. Think DSP: Digital signal processing in Python (pp. 1–11). (First edition). O’Reilly Media, Inc.[4] Thakur, B., & Mehra, R. (2016). Discrete Fourier Transform Analysis with Different Window Techniques Algorithm. International Journal of Computer Applications, 975, 8887.[5] Kong, Q., Siauw, T., & Bayen, A. (2020). Fourier Transform. Python programming and numerical methods: A guide for engineers and scientists (pp. 415–444). Academic Press.[6] Numpy Documentation, API Reference, Discrete Fourier Transform (numpy.fft). [Accessed on 2/2/2023][7] Scipy Documentation, API Reference, Legacy discrete Fourier transform (scipy.fftpack). [Accessed on 2/2/2023]Learn Discrete Fourier Transform (DFT) was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.