+41 768307656
info@htc-sagl.ch

Archive: 22/03/2023

Big Data

Population and houses growth in Switzerland

Mar 22, 2023 by Helvetica Technical Consulting 2 Comments

Switzerland is known for its high standard of living and picturesque landscapes, making it a popular destination for expats, students, and travelers. However, it is also known for its high cost of living, including housing prices. Renting a flat in Switzerland can be expensive, especially in larger cities such as Zurich, Geneva, and Basel.

The scope of this article is a study to correlate the prices of the house in francs/m2 and correlating them with population. The data used is provided by opendata.swiss and the information of this paper is free of charge.

Data Mining & Preprocessing

All data used in this study was retrieved from opendata.swiss which is the Swiss public administration’s central portal for open government data.

Several files with CSV and XLS extensions were used and adapted to provide a full dataset of information regarding population growth, buildings construction and price variation through the years.

Population data set cover 1950-2020, classified by sex, provenience & canton

Building construction dataset on the other side starts in 2003 to 2020 classified by flat or building & canton

Last set is about price per m2 in swiss francs. This set starts in 2012 until 2020 classified by canton & year of construction, from older than 1919 up to 2021. For our purpose, average through canton value was used in order to homogenize data accross years and building age.

Population data was truncated to start in 2003 to match building construction data set.

Analysis

For the analysis few statistical indicatore were used:

  • Arithmetic mean, also known as the average, is a measure of central tendency that represents the typical value of a set of numbers. It is calculated by adding up all the values in a set and then dividing the sum by the number of values in the set. The arithmetic mean is commonly used in statistics to summarize the data and to compare different sets of data. It is a useful measure of central tendency when the data is evenly distributed and does not have any extreme outliers. However, it can be influenced by outliers, and in such cases, other measures of central tendency such as the median or mode may be more appropriate. Defined as:
  • Standard deviation, The standard deviation is a measure of the amount of variation or dispersion in a set of data. It is calculated as the square root of the variance, which is the average of the squared differences of each value from the mean. The standard deviation is commonly used in statistics to describe the spread of a distribution, with a higher standard deviation indicating a wider spread of values and a lower standard deviation indicating a narrower spread of values. It is also used in inferential statistics to calculate confidence intervals and to test hypotheses about the population from which the sample was drawn. Defined as:

After calculations, graphs were constructed to visualize data and get information.

Population Data

Data recall population from year 1950 until 2020. After importing data, it is usefull to display visual information of total values both for sex and citizenship. The final graph after filtering data is as follows:

Adding a linear trend, gives that in 2030 the population will be around 9 millions.

To have further detail on population, it is possible to use population change by canton using standard deviation, to see data variation through the years.

Higher values means high variation in positive(growing) direction

Houses Data

Data about constructions in switzerland is imported. This data covers from 2003 to 2020.

It is clearly visible that the number of new construction reaches its peak in 2015 and then change its direction to the lower values.

Rent average price m2/chf

Data is categorized by canton and year, from 2012 to 2020 and the value is expressed as average through 26 cantons. Due to lowering number of new construction, one can say that prices will growth. For this reason, this dataset can be usefull to study if there are some variation in the prices. Note that this values includes existing buildings and new constructed. Original dataset considers building older that 1919 up to 2021. For practical purposes, data was filtered.

To have a better understanding, difference between 2012 and 2020 prices is summarized and plotted as follow:

Conclusion

The highest price deviation are the AI, Appenzeller Inner, second places is for BS, Basel City and third place is GL, Glarus cantons. . On the other hand, Basel Stadt has the higher variation in the prices, passing from 16.90 chf/m2 to 18.2 chf/m2.

Zurich city which has the highest population increase during the last 20 years, don’t show a proporcional increase in the price, passing 18.5 to 19.3 chf/m2.

A note from last graph is about zug that the price does not changes over 8 years, while Grisons and Schwyz the prices are lower the befor

It is worth to recall that prices are on average basis for all houses present in the canton and the price is referred only to rent, other expenses are not included like common heating, waste, cleaning, parking and other amenities.

Population-and-houses-growth-in-Switzerland

Average rent in Swiss francs according to the number of rooms and the canton | opendata.swiss

Demographic evolution, 1950-2021 | opendata.swiss

Average rent per m2 in Swiss francs according to the age of construction and the canton | opendata.swiss

Consulting

Hydrogen Generation Due to High Voltage in Cathodic Protection

Mar 8, 2023 by Helvetica Technical Consulting 0 Comments

Introduction

Cathodic protection is a widely used technique to prevent corrosion of metal structures in various industrial applications. The process involves making the metal structure cathodic with respect to a more easily corroded metal or an inert anode. This results in a flow of current, which causes the metal to be protected from corrosion. However, cathodic protection can also lead to the generation of hydrogen gas, which can cause hydrogen embrittlement.

Technical Background

When cathodic protection is applied, a voltage is applied to the metal structure, which is more negative than the equilibrium potential of the metal in the electrolyte. This negative potential causes a flow of electrons from the anode to the cathode. At the cathode, hydrogen ions are reduced to form hydrogen gas. This is a normal process in cathodic protection, but at high potentials, the amount of hydrogen generated can be excessive and lead to hydrogen embrittlement.

Hydrogen embrittlement occurs when hydrogen diffuses into the metal and interacts with the metal lattice. This can reduce the ductility and fracture toughness of the metal, making it more susceptible to cracking and failure. The severity of hydrogen embrittlement depends on factors such as the material, the level of hydrogen exposure, and the applied stress.

History

Hydrogen embrittlement was first observed in the mid-19th century in steel rails used in railway tracks. The rails were observed to fracture suddenly, even though they had not been subjected to excessive loads. It was later discovered that the rails had been exposed to hydrogen gas, which had caused them to become brittle and prone to fracture. Since then, hydrogen embrittlement has been observed in various other metals and alloys.

Mitigation Strategies

To mitigate the risk of hydrogen embrittlement in cathodic protection, several strategies can be employed. One approach is to limit the amount of hydrogen generated at the cathode by using lower cathodic potentials or adding inhibitors to the electrolyte. Another approach is to use materials that are less susceptible to hydrogen embrittlement, such as high-strength alloys or titanium.

Post-processing techniques can also be used to remove or reduce the amount of hydrogen in the material. For example, annealing or heat treatment can be used to diffuse the hydrogen out of the metal. Additionally, hydrogen diffusion barriers can be applied to prevent hydrogen from entering the metal in the first place.

Conclusion

In conclusion, cathodic protection is an effective method to prevent corrosion of metal structures, but it can also lead to the generation of hydrogen gas and subsequent hydrogen embrittlement. To mitigate the risk of hydrogen embrittlement, it is important to limit the amount of hydrogen generated at the cathode and use materials that are less susceptible to hydrogen embrittlement. Regular inspections and proactive corrosion management are crucial to detecting any signs of hydrogen embrittlement or other types of corrosion damage.

References:

  • H. Wang, J. Zheng, Q. Zhang, & Y. Wei. (2019). Mitigation of hydrogen embrittlement of a 7B04 aluminum alloy by controlling the microstructure. Materials & Design, 170, 107675.
  • M. W. Kendig & R. G. Buchheit. (2003). Hydrogen embrittlement. Corrosion: Understanding the Basics, 305-324.
  • J. R. Scully & H. Zhu. (2010). Hydrogen embrittlement and hydrogen-induced cracking. ASM Handbook, 13B, 1085-1101.
Machine Learning

Sentiment Analysis, how to sell more and better

Mar 6, 2023 by Helvetica Technical Consulting 0 Comments

Sentiment analysis is a powerful tool that can be used to analyze and understand the emotions, opinions, and attitudes expressed in a text or speech. This technology has gained significant importance in recent years as it helps businesses to understand customer feedback and sentiment, which can ultimately help them to make better decisions.

In this article, we will explore the techniques used in sentiment analysis and how it can help the retail industry.

What is Sentiment Analysis?

Sentiment analysis, also known as opinion mining, is a process that uses natural language processing, machine learning, and other computational techniques to identify and extract subjective information from text or speech data. It involves classifying the sentiment of a piece of text into positive, negative, or neutral categories.

The techniques used in sentiment analysis can vary from simple rule-based methods to more advanced machine learning algorithms. The most common approach is to use a combination of both methods.

Techniques used in Sentiment Analysis

Rule-based Methods
Rule-based methods rely on a set of predefined rules to classify sentiment in text. These rules can be based on specific words, phrases, or patterns that are associated with a particular sentiment. For example, if a sentence contains words like ‘good,’ ‘great,’ or ‘excellent,’ it is likely to be classified as positive.

While rule-based methods are simple and easy to implement, they can be less accurate than more advanced machine learning algorithms. They also require constant updating as language and expressions change over time.

Machine Learning
Machine learning algorithms use statistical models to learn from data and make predictions. These algorithms require a large dataset of labeled examples to train the model. The labeled data consists of text or speech samples, along with their corresponding sentiment labels.

There are several types of machine learning algorithms used in sentiment analysis, including:

Naive Bayes: This algorithm uses probabilistic models to classify text based on the frequency of words in the document.
Support Vector Machines (SVM): SVM is a supervised learning algorithm that can classify text into two or more categories based on the features extracted from the text.
Deep Learning: Deep learning techniques such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) can be used to learn complex patterns and relationships in text data.
How Sentiment Analysis Can Help the Retail Industry

The retail industry is one of the most competitive industries in the world, and understanding customer feedback and sentiment is crucial for success. Sentiment analysis can help retailers in several ways:

Customer Feedback Analysis

Sentiment analysis can be used to analyze customer feedback and reviews from various sources such as social media, review sites, and customer surveys. This information can help retailers to identify areas of improvement and make changes accordingly. For example, if customers are complaining about the long checkout lines, retailers can take steps to improve the checkout process and reduce wait times.

Product Development

Sentiment analysis can be used to analyze customer feedback on existing products and services. Retailers can use this information to identify areas for improvement or to develop new products that better meet the needs of their customers.

Brand Management

Sentiment analysis can be used to monitor brand reputation and identify potential issues before they become major problems. By analyzing social media conversations and other online content, retailers can track customer sentiment and respond quickly to any negative feedback.

Customer Service

Sentiment analysis can be used to monitor customer service interactions and identify areas where improvements can be made. By analyzing customer feedback and sentiment, retailers can improve the quality of customer service and enhance the overall customer experience.

Conclusion

Sentiment analysis is a powerful tool that can help the retail industry to understand customer feedback and sentiment. By analyzing customer feedback and sentiment, retailers can identify areas for improvement, develop new products, and enhance the overall customer experience. With the increasing availability of data and the advancements in machine learning algorithms, sentiment analysis has become more accurate and efficient. Retailers can now use sentiment analysis tools to process vast amounts of customer feedback and sentiment data in real-time, enabling them to make better business decisions quickly.

In addition to the benefits discussed above, sentiment analysis can also be used to analyze competitor data, track industry trends, and identify emerging market opportunities. By leveraging sentiment analysis, retailers can gain valuable insights into their customers’ needs and preferences, allowing them to stay ahead of the competition and adapt to changing market conditions.

However, it is important to note that sentiment analysis is not a perfect tool and can be affected by biases and inaccuracies in the data. For example, sarcasm and irony can be challenging to detect, and sentiment analysis tools may struggle to identify subtle nuances in language and context.

To mitigate these challenges, retailers should use a combination of sentiment analysis tools and human analysis to ensure the accuracy and relevance of their insights. By combining the power of technology with the expertise of human analysts, retailers can gain a deeper understanding of customer sentiment and make better decisions that drive business growth.

In conclusion, sentiment analysis is a powerful technology that can help the retail industry to understand customer sentiment and feedback. By leveraging sentiment analysis, retailers can gain valuable insights into their customers’ needs and preferences, allowing them to make better decisions and improve customer satisfaction.

Machine Learning

How Machine Learning can save money in tomorrow’s industry

Mar 6, 2023 by Helvetica Technical Consulting 0 Comments

The oil and gas industry is constantly looking for ways to improve the efficiency of its operations and reduce downtime. One way to achieve this is through predictive maintenance, which uses machine learning algorithms to identify potential equipment failures before they occur. This article will explore the advantages and disadvantages of using machine learning for predictive maintenance in the oil and gas industry.

Advantages of Machine Learning for Predictive Maintenance in the Oil and Gas Industry

  1. Improved Equipment Performance

Machine learning algorithms can analyze real-time data to predict equipment failures before they occur. This allows companies to schedule maintenance proactively, resulting in improved equipment performance and reliability. By predicting when maintenance is needed, companies can avoid costly and time-consuming downtime.

  1. Reduced Downtime

Downtime can be costly for any industry, but it is especially significant in the oil and gas industry, where every minute of downtime can result in lost revenue. Predictive maintenance can help reduce downtime by identifying potential equipment failures before they occur. By preventing equipment failures, companies can avoid the costly downtime associated with repairs.

  1. Increased Safety

Predictive maintenance can help improve safety by identifying potential equipment failures before they occur. This can help prevent accidents and improve overall safety in the workplace. By identifying potential safety hazards, companies can take proactive steps to prevent accidents and keep their employees safe.

  1. Reduced Maintenance Costs

Predictive maintenance can help reduce maintenance costs by allowing companies to schedule maintenance only when it is necessary. This can reduce the need for unnecessary maintenance, which can be costly. By predicting when maintenance is needed, companies can avoid costly repairs and reduce their overall maintenance costs.

Disadvantages of Machine Learning for Predictive Maintenance in the Oil and Gas Industry

  1. Cost

Implementing machine learning algorithms can be expensive, especially for small to medium-sized companies. It may require significant investment in new hardware and software, as well as training for employees. However, the cost of implementing machine learning for predictive maintenance must be weighed against the potential cost savings from reduced downtime and maintenance costs.

  1. Data Quality

Predictive maintenance relies on high-quality data to produce accurate results. If the data is incomplete or inaccurate, the algorithms may produce inaccurate predictions, leading to unnecessary maintenance or equipment failure. To overcome this challenge, companies must ensure that they have high-quality data and that their algorithms are properly calibrated.

  1. Complexity

Machine learning algorithms can be complex, and it may be difficult for non-experts to understand how they work. This can make it challenging to implement and maintain the algorithms. To overcome this challenge, companies must ensure that they have skilled data analysts who can develop and maintain the algorithms.

  1. Need for Skilled Data Analysts

Machine learning algorithms require skilled data analysts to develop and maintain them. These experts can be difficult to find and expensive to hire, which can be a significant barrier to implementing predictive maintenance. Companies must ensure that they have the necessary resources to develop and maintain their algorithms.

Conclusion

The advantages of using machine learning for predictive maintenance in the oil and gas industry are clear. Improved equipment performance, reduced downtime, increased safety, and reduced maintenance costs can all be achieved through the use of these algorithms. However, the disadvantages, such as cost, data quality, complexity, and the need for skilled data analysts, must also be considered. Overall, the benefits of using machine learning for predictive maintenance in the oil and gas industry outweigh the drawbacks, making it an attractive option for companies looking to improve their operations. To successfully implement predictive maintenance using machine learning, companies must ensure that they have the necessary resources and expertise to develop and maintain their algorithms.

Big Data

Data Scientist: Current State and Future Trend, a new role for the future

Mar 6, 2023 by Helvetica Technical Consulting 0 Comments

The field of data science has exploded in recent years, with demand for skilled professionals at an all-time high. Universities around the world now offer a variety of courses and degree programs in data science, including both undergraduate and graduate options. Online learning platforms such as Coursera, Udacity, and edX offer massive open online courses (MOOCs) in data science, allowing individuals to gain valuable knowledge and skills without enrolling in a full-time program.

While many universities and online courses provide a solid foundation in data science, heuristic knowledge gained through practical experience is equally important. Data scientists must have strong programming skills, as well as expertise in statistical analysis, machine learning, and data visualization. Effective communication skills are also essential, as data scientists must be able to explain their findings to both technical and non-technical stakeholders.

Some minimum requirements for a career in data science include a bachelor’s degree in a related field such as computer science, statistics, or mathematics, as well as experience with programming languages such as Python or R. However, many employers now require advanced degrees and significant work experience in the field.

According to the Bureau of Labor Statistics, the demand for data scientists is projected to grow by 16% between 2020 and 2030. This growth is expected to be driven by increasing demand for data-driven decision-making across industries. The field of data science is continually evolving, and professionals must keep up with the latest developments and technologies to stay competitive.

In addition to traditional data science roles, there are also emerging areas of specialization within the field, such as data engineering, data visualization, and data journalism. These specializations offer opportunities for individuals to focus on specific aspects of data science and develop expertise in a particular area.

In conclusion, data science is a rapidly growing field with strong demand for skilled professionals. While universities and online courses provide a foundation, practical experience and heuristic knowledge are equally important. Effective communication and programming skills are essential, and advanced degrees and work experience are increasingly required. With the continued growth of data-driven decision-making, the demand for data science professionals is expected to remain high.

Big Data

Key Distinctions between Scientists and Engineer, to empower Data Analytics

Mar 6, 2023 by Helvetica Technical Consulting 0 Comments

Data analytics is a growing field, where data scientists and engineers are crucial for its success. Both roles involve working with data, but have distinct responsibilities. Science is more like research, while data engineering is more like development. The first analyze data to extract insights and make predictions, while data engineers design and maintain systems to enable data scientists to work with data.

Data scientists ask the right questions and find meaningful insights from data, while data engineers build and maintain the infrastructure. Engineering involves building the infrastructure to support data science, while data science involves using that infrastructure to extract insights to make data usable, while data science makes sense of it.

Both data scientists and data engineers have strong employment prospects. The demand for data scientists is projected to grow by 16% between 2020 and 2030, and for computer and information technology occupations, which include data engineers, by 11%. The increasing importance of data-driven decision making across industries means that the demand for both roles will continue to rise.

If you want to become a data engineer or data scientist, there are various educational paths to take. Many universities offer undergraduate and graduate programs in data science, computer science, or related fields. Additionally, various online courses and bootcamps offer training in data analytics, machine learning, and other relevant skills.

Data science and data engineering have vast and varied applications. In healthcare, data analytics improves patient outcomes and streamlines processes. In finance, data analytics detects fraud and predicts market trends. In retail, data analytics personalizes marketing campaigns and optimizes supply chain operations. Data science and data engineering drive innovation and create value across industries.

Conclusion

In conclusion, data scientists and data engineers are critical for data analytics success, with essential, distinct responsibilities. The demand for both roles will continue to increase, as data-driven decision making becomes more important. Pursuing a career in data analytics offers various educational paths and fields of application to explore.

Further resources

  1. “Python Data Science Handbook” by Jake VanderPlas: https://jakevdp.github.io/PythonDataScienceHandbook/
  2. “Data Science Essentials” by Microsoft: https://docs.microsoft.com/en-us/learn/paths/data-science-essentials/
  3. “Data Engineering Cookbook” by O’Reilly Media: https://www.oreilly.com/library/view/data-engineering-cookbook/9781492071424/
  4. “Data Science for Business” by Foster Provost and Tom Fawcett: https://www.amazon.com/Data-Science-Business-data-analytic-thinking/dp/1449361323
  5. “Data Engineering on Google Cloud Platform” by Google Cloud: https://cloud.google.com/solutions/data-engineering/
  6. “Applied Data Science with Python” by Coursera: https://www.coursera.org/specializations/data-science-python
Machine Learning

Supervised, Unsupervised & Reinforced Learning, a quick intro!

Mar 1, 2023 by Helvetica Technical Consulting 0 Comments , , , ,

In the field of predictive maintenance for rotating equipment, machine learning algorithms can be classified into three categories: supervised learning, unsupervised learning, and reinforced learning. Each of these approaches has its strengths and weaknesses, and choosing the right approach depends on the nature of the problem at hand. In this essay, we will explore the differences between these approaches and their applications in the context of predictive maintenance for rotating equipment.

Supervised Learning

Supervised learning involves training a model on labeled data, where both the input data and the desired output are provided. The goal is to learn a function that can predict the output for new, unseen input data. In the context of predictive maintenance for rotating equipment, supervised learning can be used to predict the remaining useful life of a machine or to detect anomalies that may indicate the onset of a fault.

One common application of supervised learning in predictive maintenance is to analyze vibration data from rotating machinery. By training a model on labeled data that indicates when a fault occurred and the corresponding vibration patterns, the algorithm can learn to identify these patterns in real-time data and predict potential faults before they occur.

Unsupervised Learning

Unsupervised learning involves training a model on unlabeled data, where the input data is provided without any corresponding output. The goal is to find patterns or structures in the data that can be used to make predictions or identify anomalies. In the context of predictive maintenance for rotating equipment, unsupervised learning can be used to identify patterns or clusters in sensor data that may indicate the presence of a fault.

One common application of unsupervised learning in predictive maintenance is to use clustering algorithms to group similar data points together. By analyzing the clusters, it may be possible to identify patterns that are indicative of a specific type of fault or to detect anomalies that may indicate the onset of a fault.

Reinforced Learning

Reinforcement learning involves training a model to make decisions based on feedback from the environment. The goal is to learn a policy that maximizes a reward signal over time. In the context of predictive maintenance for rotating equipment, reinforced learning can be used to develop maintenance schedules that minimize downtime and reduce costs.

One common application of reinforced learning in predictive maintenance is to use a model to determine when maintenance should be performed based on the condition of the machine and the cost of downtime. By learning a policy that balances the cost of maintenance with the cost of downtime, it may be possible to develop a more efficient maintenance schedule that reduces costs and increases efficiency.

Choosing the Right Approach

The choice of machine learning approach depends on the nature of the problem at hand. Supervised learning is best suited for problems where labeled data is available, and the goal is to predict an output for new, unseen data. Unsupervised learning is best suited for problems where the data is not labeled, and the goal is to identify patterns or anomalies in the data. Reinforced learning is best suited for problems where the goal is to develop a policy that maximizes a reward signal over time.

In the context of predictive maintenance for rotating equipment, a combination of these approaches may be used to develop a comprehensive predictive maintenance strategy. For example, supervised learning can be used to predict the remaining useful life of a machine, unsupervised learning can be used to identify patterns or clusters in sensor data, and reinforced learning can be used to develop a maintenance schedule that balances the cost of maintenance with the cost of downtime.

Conclusion

In conclusion, machine learning algorithms can be classified into three categories: supervised learning, unsupervised learning, and reinforced learning. Each of these approaches has its strengths and weaknesses, and choosing the right approach depends on the nature of the problem at hand. In the context of predictive maintenance for rotating equipment, a combination of these approaches may be used to develop a comprehensive predictive maintenance strategy that minimizes downtime, reduces costs

Consulting

Hydrogen from Ammonia, a fuel for the future

Feb 28, 2023 by Helvetica Technical Consulting 0 Comments , , , ,

Green ammonia is an emerging technology that has the potential to revolutionize the production of hydrogen and significantly reduce carbon emissions. In this article, we will discuss the production of hydrogen from green ammonia, key production and money figures, companies involved, and future trends.

Production of Hydrogen from Green Ammonia

Green ammonia is produced by using renewable energy sources such as wind or solar power to power the Haber-Bosch process, which produces ammonia. Green ammonia can then be used as a feedstock for the production of hydrogen through the process of ammonia cracking. The reaction is endothermic, requiring a reactor heated to a high temperature of around 700-900°C to break down ammonia into its constituent elements, nitrogen and hydrogen.

Key Production and Money Figures

The production of hydrogen from green ammonia has several advantages over traditional methods, including zero carbon emissions and lower energy requirements. According to the International Energy Agency (IEA), the production of green ammonia is expected to reach 25 million tonnes by 2030 and 500 million tonnes by 2050. The IEA also estimates that the production of green ammonia could reduce the cost of producing hydrogen by up to 50% compared to traditional methods.

Companies Involved

Several companies are involved in the production of green ammonia, including Yara, the world’s largest producer of ammonia, and Siemens Energy, which has developed an electrolysis-based process for producing green ammonia. Other companies involved in the production of green ammonia include Ørsted, a leading renewable energy company, and Air Liquide, a global leader in industrial gases.

Future Trends

The future of green ammonia production looks bright, with the potential for significant growth and contribution to reducing carbon emissions in the energy and agricultural sectors. The IEA has identified green ammonia as a key technology that could help to reduce carbon emissions. Green ammonia has the added benefit of being used as a fertilizer, further reducing the carbon footprint of agriculture. In addition, the use of green ammonia in the shipping industry as a fuel is being explored as a potential replacement for fossil fuels.

Conclusion

Green ammonia is a promising technology that has the potential to revolutionize the production of hydrogen and significantly reduce carbon emissions. Key production and money figures suggest that the production of green ammonia could increase significantly over the next few decades, with the potential to reduce the cost of producing hydrogen by up to 50%. Several companies are involved in the production of green ammonia, and the future looks bright with the potential for significant growth and contribution to reducing carbon emissions in the energy and agricultural sectors.

Big Data

Which Is The Difference Between Data Scientist And Data Engineer?

Feb 28, 2023 by Helvetica Technical Consulting 0 Comments , , ,

Data scientist and data engineer are both essential roles in the field of data analytics, but they have distinct responsibilities. According to Max Shron in “Thinking with Data: How to Turn Information into Insights,” “data science is more like a research project, while data engineering is more like a development project.” This means that while data scientists focus on analyzing data to extract insights and make predictions, data engineers are responsible for designing and maintaining the systems that enable data scientists to work with the data.

Andreas Müller and Sarah Guido echo this sentiment in “Introduction to Machine Learning with Python: A Guide for Data Scientists,” stating that “data scientists are concerned with asking the right questions and finding meaningful insights from data. Data engineers are responsible for designing and maintaining the systems that enable data scientists to work with the data.” DJ Patil and Hilary Mason similarly note in “Data Driven: Creating a Data Culture” that “data engineering involves building the infrastructure to support data science, while data science involves using that infrastructure to extract insights from data.”

Joel Grus adds in “Data Science from Scratch: First Principles with Python” that “data engineering involves building the infrastructure to support data science, while data science involves using that infrastructure to extract insights from data.” Finally, Martin Kleppmann sums it up in “Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems” by saying that “data science is about making sense of data, while data engineering is about making data make sense.”

In summary, data scientists focus on extracting insights from data, while data engineers focus on building the infrastructure to store and process that data. While there may be some overlap between the roles, they have distinct responsibilities and focus on different aspects of working with data. Both roles are crucial in modern data-driven organizations, and they often work together closely to achieve common goals

Big Data

Rock Music is Alive and Powerful! Statistics from 1950 and 2020

Jan 11, 2023 by Helvetica Technical Consulting 5 Comments , ,

This article was done to get some statistics about rock music and what big data analysis can do to gather or discover hidden useful information.

The following analysis gets the data from Kaggle, free license

What is Kaggle? According to online definitions, Kaggle, a subsidiary of Google LLC, is an online community of data scientists and machine learning practitioners. inside the website can be found courses, datasets, contest/challenges including money.

Dataset can be uploaded by single usernames or by companies during a competition.

 Scope of the Study

A lot of considerations can be made from the history of rock music, but the scope of this study is to support the changes that music rock did during the years.

Rock music, as an alternative of pop music (intended as common or soft) in the beginning was an underground music that gained fame during the years, with a constant increase. Some people or critics claim that rock is dead, but we will seek if there is a truth on this sentence.

  Data

Dataset is from 2020 retrieved from spotify covering rock songs from 1950 to 2020 with 5484 songs and 17 tags/label to identify and classify a song. From the tag list, only popularity is an index from the audience feedback while the remaining tags describe the song characteristics.

  1. Index
  2. Name: Song’s name
  3. Artist
  4. Release date
  5. Length: in minutes
  6. Popularity: A value from 0 to 100
  7. Danceability: Describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity.
  8. Acousticness: A confidence measure from 0.0 to 1.0 of whether the track is acoustic.
  9. Energy: Represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale.
  10. Instrumentalness: Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”.
  11. Key: The estimated overall key of the track. Integers map to pitches using standard Pitch Class notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on.
  12. Liveness: Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live.
  13. Loudness: The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks.
  14. Speechiness: This detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value.
  15. Tempo: The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
  16. Time Signature: An estimated overall time signature of a track. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure).
  17. Valence: Describes the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).

Popularity requires some clarification from analytical point of view and need some assumptions. We don’t know when the popularity was measured, monthly or yearly, and also in which year. Considering this lack of information, we will assume likelihood that popularity was calculed in 2020 when considering songs from 1950 to 2019.

Data Pre-processing & Feature Engineering

After loading the data, we need to manipulate it according to our scope of the study, more specifically we will count the letters both in the artist’s name and song’s name.

The name of the song contains some noise created by the versions mastered or remastered. this creates a distortion in the real name of the song. Most of time, remastering a song has the only effect to clean using new technologies and also to refresh the mind of people.

Since there are 5848 rows in the data, this creates a lot of noise, so the best way for filtering data, is to preprocesssing in aggregated way following statistical parameters, mean, max & min of the values for each year from 1956 to 2020. This leads to a new data set of 65 rows where every row is one year.

Below you can find complete pdf.

historyofrock