September 2018

  1. home
  2. blog
  3. 2018
  4. September
Machine Learning: autonomous learning

Machine Learning: autonomous learning

by Sílvia Raposo
2018-09-21
3 min

Machine Learning has been developing further every day, thanks to the digital transformation movement. The original basis was a theory that believed computers could learn to perform specific tasks and to recognise patterns. The challenge was simple: to check if computers could learn from data.

Machine Learning provides systems with the possibility to learn and improve from experience, without needing specific programming for that effect. The focus is on developing programs that use available data and can learn on their own. The mathematical models are built and powered with – potentially – large amounts of data. The algorithms learn to identify patterns and to extract insights that are applied when new information is processed. This term dates back to 1959, when the pioneer Arthur Samuel defined Machine Learning as the ability of a computer to learn without being explicitly programmed to do so.

This learning process starts with data processing and trying to identify patterns. The main goal is to allow computers to learn autonomously without the need for human assistance, using that knowledge to make decisions according to what was “learnt”. Even though machine learning algorithms have been around for a long time, the application of these mathematical calculations to Big Data, with more frequency is a recent development. However, according to industry reports, what is considered to be an exponential growth in this area today is going to be seen as only “baby steps” in 50 years. This AI field is expected to grow extremely fast in the coming years.

Examples of Machine Learning

The continuing interest in this practice stems from a few key factors that have also made data mining and Bayesian analysis extremely popular: growth in the volume and variety of available data; cheaper and more powerful computational processes; and low cost storage.

A few examples of machine learning applications in some companies include self-driving vehicles; recommendations from online platforms such as Amazon and Netflix based on users’ behaviour; voice recognition systems such as SIRI and Cortana; PayPal’s platform, which is based on machine learning algorithms to fight fraud by analysing large quantities of data from the customer and assessing risks; the model from Uber that uses algorithms to determine time of arrival and departure locations; SPAM detecting mechanisms in email accounts; facial recognition that occurs in platforms such as Facebook.

Industries that are choosing Machine Learning

Most industries with large amounts of data have already acknowledged the potential of this technology. The possibility to extract insights allows companies to obtain a competitive advantage and work more efficiently.

Financial Services

Banks and other financial entities are using machine learning with two goals: extracting valuable insights from customer data and preventing fraud. Insights identify investment opportunities according to customers’ profiles, and, concerning fraud, the identification of high-risk customers and suspect transactions is improved.

Furthermore, this technology can also influence customer satisfaction. By analysing a user’s activity, smart machines can predict, for example, a possible account closure before it happens and prompt mitigating actions.

Health

Health entities can capitalise on the integration between IoT and data analysis to develop better solutions for patients. The emergence of wearables allows acquiring data related to the patients’ health, which, in turn, allows health professionals to detect relevant patterns including risk patterns. Therefore, this technology offers the potential for better diagnosis and treatment.

Retail

Nowadays, the impact of smart machines in users’ retail experience is quite obvious. The result is a highly personalised service that includes recommendations based on purchase history or online activity; improvements in customer service and delivery systems, where machines decipher the meaning of users’ emails and delivery notes, in order to prioritise tasks and ensure customer satisfaction; and dynamic price management by identifying patterns in price fluctuations and allowing to prices to be determined according to the demand. The ability to gather, analyse and use data to personalise, for example, a purchase experience (or implement a marketing campaign) is the future of retail.

Transportation

Analysing data to identify patterns and trends is key to the transportation industry, since profit growth means more efficient routes and the projection of potential problems. Data analysis and the modelling aspects of machine learning are important tools for delivery companies and public transportation, allowing them to improve their income.

Machine learning apps allow companies to automate the analysis and interpretation of business interactions, extracting valuable insights that make personalising products and services, possible.

Xpand IT has a complete service portfolio in Machine Learning. If you want to know how to use Machine Learning in your business and obtain real added value, we can help. Do you want to know how we can help your business? Contacts us here and get the best out of this technology!

Sílvia RaposoMachine Learning: autonomous learning
read more
Apache Superset Open Source BI: almost the alternative to Tableau

Apache Superset Open Source BI: almost the alternative to Tableau

by Susana Santos
2018-09-17
4 min

What is Apache Superset?

Superset is a modern BI app with a simple interface, feature-rich when it comes to views, that allows the user to create and share dashboards.

This app is simple and doesn’t require programming, and allows the user to explore, filter and organise data. The best part is… it’s Open Source!

What does Apache Superset provide?

What is truly appealing about Apache Superset is the fact that you can explore each dashboard in a complex way. Superset allows you to focus on each graph/metric and easily filter and organise.

Another attractive feature in this app is the SQL/IDE editor with interactive querying.

Concerning security, Superset allows you to define a list of users and a list of default functionalities (associated with the groups of users) and allows you to view user statistics, providing you total control. You can establish baseline permissions, as well as granting access to certain views or menus. The app also provides an action log.

Visually, Superset has a minimalist and well-organised interface. Even though it is not as easy to use as Tableau, Superset can be an alternative to creating dashboards or people with some knowledge of SQL.

Database support

Superset supports most SQL databases by using Python ORM (SQL Alchemy), which allows you to access MySQL, Postgres, Oracle, MS SQL Server, MariaDB, Sybase, Redshift and others (more information here).

Superset also works with Druid (for example, Airbnb uses Superset with Druid 0.8x), but it does not have all the advanced features available.

SQL-LAB

This feature is definitely a plus. SQL-Lab allows you to select a database, schema and table (previously uploaded) and do an interactive query, preview the data and also save the query history (as shown below).

SQL Lab

A semantic layer allows you to define fields and metrics (for example, ratios or anything expressed by SQL):

SQL Lab

Query history:

Query history
Query history

You also have Python modules available (some available macros), inside SQL, via Jinja.

The least positive side of this is the fact that you cannot add or query multiple tables at the same time. The solution is making a view, which works as a logical layer and abstracts the query from SQL, therefore acting as a virtual table. The only negative aspect of this is that there will always be a query running against another view query, thereby potentially resulting in performance issues.

How to create a dashboard

To create a dashboard, Superset works as follows: there are sources, where you can find databases and tables; slices which are sheets with graphs; and, lastly, dashboards which are composed of groups of slices. Each slice is associated with one or more dashboards, and each dashboard has various associated slices.

Apache Superset dashboard

Views have different types of graphs available such as histograms, box plots, heatmaps or line charts.

Apache Superset dashboard

It is simple to edit graphs: the available features for each view are on the left-hand side, and you just have to change them and press “Run Query”.

Although flexible in most areas, Superset imposes some standardisation, which happens with the colour schemas.

Apache Superset dashboard

Each view allows you to filter views through wildcards.

Apache Superset dashboard

Superset also allows you to share the view, export data to .json and .csv, and see the exact query performed behind each view.

Apache Superset dashboard

Security

Superset integrates with the main authentication backends (database, OpenID, LDAP, OAuth, REMOTE_USE, …).

Concerning privileges, as stated above, this app provides default roles such as Admin (full access), Alpha Gamma, Sql_lab and Public.

It is possible to establish permissions for each user, restricting access to a subset of data sources, menus, views, specific metrics and other criteria. Hence, it is relatively easy to define which type of permission and/or access to data is granted to each person.

People using Superset

According to GitHub, Superset is currently being used by Airbnb, Twitter, GfK Data Lab, Yahoo!, Udemy and others.

It is important to note that “Superset was tested in large environments with hundreds of users. The production environment of Airbnb runs with Kubernetes and more than 600 active users who see more than 100 thousand graphs per day”.

Superset Vs Tableau

Tableau

Superset

  • Able to join between tables within the same or different DBs.
  • Unable to query/join multiple tables. Only possible view by view, which means having multiple queries, thereby affecting performance.
  • Detailed customisation of dashboards, with legends, filters, tags, etc.
  • Limited customisation by type of view (however, creation of CSS templates is available).
  • Easy beginner learning and doesn’t require users to know SQL. Since the platform allows more complex and flexible tasks, there is a second learning curve for users who want to make the best use of Tableau.
  • Easy and smooth learning, but requires SQL knowledge from users.
  • Paid
  • Free and Open Source

Superset’s main advantages

Besides all the advantages already stated, one of the main features of Superset is… it’s Open Source Business Intelligence!

Other advantages:

  • Provides BI without needing code (easy to use for those who are not programmers: you only need to know basic SQL);
  • Easy and quick setup;
  • Provides “SQL-Lab” that allows interactive querying;
  • A semantic layer that broadens the dashboard with ratios and other metrics (based on SQL);
  • Easy and attractive interactive view, that allows data exploration;
  • Satisfies the needs of most companies to allow simple data analysis.

Superset’s disadvantages

  • The app still doesn’t support NoSQL databases;
  • Even though the number of users is growing, it still has little or no support;
  • Sometimes, SQL-Lab freezes in queries for large amounts of data;
  • Has a considerable number of other unsolved issues.

Susana Santos

Data Scientist, Xpand IT

Susana SantosApache Superset Open Source BI: almost the alternative to Tableau
read more
Pledge 1%: Supporting Associação Crescerbem with Jira Software

Pledge 1%: Supporting Associação Crescerbem with Jira Software

by Ana Paneiro
2018-09-10
3 min

Associação Crescerbem is a Private Social Solidarity Institution, founded in 2011, and its main goal is to help families of hospitalised children in economic deprivation. The association started in Dona Estefânia Hospital in Lisbon, where the headquarters still remain nearby. Meanwhile, the support has been extended to Santa Maria Hospital and Beatriz Ângelo Hospital.

The mission of Crescerbem is, specifically, to provide for families in a personalised manner, according to the specific needs of each case and helping them become more independent and autonomous. This way, follow-up does not happen only when a child is hospitalised, but also in the period after the medical discharge.

The association started offering home support and, through that, other needs started to be identified besides medical follow-up; from that point on, countless parallel projects came to life, such as the solidarity pantry (food baskets provided to families), the laundry service, and the solidarity pharmacy (which provides the necessary medication).

Despite the number of families helped and the number of existing cases, all the information was still offline. This means that it was impossible to access the necessary information without physically being in the association’s headquarters. Volunteers spent hours searching for case files and updating them, which resulted in a huge lack of visibility on the current state of each support case. Therefore, computerising all this information was critical. However, there was a problem: the lack of financial means to spend on technological solutions that could put an end to this problem.

It was at this stage that Donate IT – a community of volunteers who work with information technology – came into action through Sofia Neto, community volunteer and Collaboration & Development Solutions Lead from Xpand IT. Sofia managed to unite her work at Donate IT to the movement Pledge 1% – a movement which Xpand IT joined in 2017, thereby committing to annually donate 1% of profit and 1% of products to social solidarity institutions. Completely pro bono, Xpand IT made Jira Software available to Crescerbem (including implementation and support), thereby accomplishing Donate IT’s vision  – helping to help others  – and Pledge 1%’s vision.

Besides being extremely important to the computerisation of all documents of Crescerbem, this project proves that Jira Software has a lot more uses than software management. In practice, it can be adapted to any reality.

Inside the association, each family corresponds to an ‘issue’ in Jira. When support to a new family starts, a new issue is created and the type family is chosen. Each issue has the complete information that characterises each family: information including when the support started, how many children, which country the family is from and the contact details of the parents.

Everything is organised in different tabs in order to make viewing and editing of the information on each family easier, and all initiatives are considered subtasks: e.g. home support or medication provided. Therefore, a new registry of everything that happens has become an integral part of the process, and everyone with access permission can access that information anywhere.

Xpand IT is already working on the new release and it will have two goals: the first one is to introduce Confluence to further facilitate the sharing of information about tasks to be undertaken or families with open cases; the second one is to include Xporter, so that the association can easily export a report, for example, and to always be able to present it with the best formatting.

With this software, we have the ability to computerise all social processes and this means that access to all information related to the families is only a click away. What is the importance of this change? We will have more time. Time to begin expanding to other hospitals; time to help more families; and time to move forward with the idea of a social business, which will make Crescerbem self-sustaining.

Isabel Ramos - Co-founder, Crescerbem
Ana PaneiroPledge 1%: Supporting Associação Crescerbem with Jira Software
read more
DevOps is not Dev & Ops – What I didn’t know about DevOps

DevOps is not Dev & Ops – What I didn’t know about DevOps

by Sílvia Raposo
2018-09-06
2 min

All these years I have heard about DevOps, but I was truly convinced it was too techy for me.

I thought it was about continuous integration, automation, and awesome DevOps guys, who knew not only how to develop software but also how to release and manage production environments…

Now, I realise that I was completely wrong… DevOps is not Dev & Ops teams together… but an entire organisation that collaborates – really collaborates.

Of course you need automation; of course you need continuous integration – but that’s not all.

In a DevOps culture you must follow these rules:

  • Know the flow = understand how work goes from “to do” to “done”
  • You don’t work in a silo = instead of working in an isolated team that is just worried about their “own” work, you work for a purpose/value
  • You are constantly learning & improving = Don’t waste time – if something needs to be changed, change it

But how can you transform a whole organisation? Below, you can see some practical tips:

  • Make your work visible to everyone; don’t worry what others may think about it.
  • Change your mindset. Let me tell you a story, that someone once told me:

JFK, once when visiting NASA, saw a janitor cleaning the floor and asked him: What are you doing? He expected an answer like “I am cleaning the floor”, but instead the man said “I’m helping the men get to the Moon.”

  • Add value to your user stories; don’t create them just for someone to do something, but because you need to generate value, like improving customer satisfaction to 80%.
  • Collaborate, collaborate, collaborate even more… No man is an island, so don’t work like one.

Tools are not the most important element, but they can definitely help. Running shoes don’t make you a runner, but they will help you to run better.

If you are searching for tools that can help you understand the flow of work, make your work visible, and help you collaborate better with your team, just take a look at Jira, which allows teams to capture and organise work, assign it to the team and track team activity.

Sofia Neto

Collaboration & Development Solutions Manager, Xpand IT

Sílvia RaposoDevOps is not Dev & Ops – What I didn’t know about DevOps
read more
Xpand IT receives Atlassian Philantropy Partner of the Year 2018

Xpand IT receives Atlassian Philantropy Partner of the Year 2018

by Sílvia Raposo
2018-09-04
2 min

Barcelona, 4 September 2018 – “Moments like these are the ones that remind us that the path of giving back to the community can really make the change for a better world.” These were the words of Pedro Gonçalves, Xpand IT’s Co-Founder, after having received the award for Philanthropy Partner of The Year at the Atlassian Summit 2018.

Xpand IT was the first Portuguese company to join the Pledge 1% movement and by doing so, has committed to donate 1% of product and annual profit to charitable organisations. During the last year, it has been deeply involved with the philanthropic movement, developing a major series of initiatives aimed at giving back to the community.

„Atlassian is thrilled to recognize and honor our 2018 Partner Award recipients„, said Martin Musierowicz, Atlassian’s Head of Global Channels. „Solution Partners are instrumental to our customers‘ success and we are excited to be able to highlight some of our top partners who are going above and beyond to support customers and provide Atlassian services.„

Xpand IT plans to raise the bar in disseminating the Pledge 1% mission in 2019 and multiply the number of initiatives aimed at helping those in need – all the while improving companies’ success through better collaboration using Atlassian technology.

Atlassian is the company behind products such as Jira, Jira Service Desk, Bitbucket and Confluence. Their mission is to help every team  unleash their potential. Xpand IT has achieved highest Solution Partner Level, Platinum, and has an impressive track-record of implementing projects based on Atlassian products.

For Sofia Neto, Collaboration & Development Solutions Lead at Xpand IT, being present at the Summit is not only a unique opportunity to meet people and share know-how and experiences, but is also recognition of the continuous hard work: “We’ve had the opportunity to participate in such exclusive events and the experience goes far beyond a traditional one. This is the second year in a row that we have been distinguished by Atlassian, this time recognising our involvement in the philanthropic movement Pledge 1%, and it’s something that truly makes us proud and just makes us want to do more and more. This is definitely a huge part of what it is to be an Xpander.”

Xpand IT team receives award from Mike Cannon-Brookes, the co-founder of Atlassian, the Philantropy Partner of the year 2018, in Barcelona.
Sílvia RaposoXpand IT receives Atlassian Philantropy Partner of the Year 2018
read more

Search

Popular Posts