Category: Architecture & Developing

Tip: Java 8 Streams flatmap map lists

A little example, I had to develop as a teaching use case, of how to take a map with Integer keys and values with a List as value and find a specific value in any List in the whole map using Java Streams.

boolean b1 returns true because the element exists
boolean b2 is the response to an element that doesn’t exist

Go Language, First Steps

Go, first experience.

What

We are living in a very rich IT ecosystem, where developers aren’t afraid to explore new languages, tools and to use out of the box thinking. It’s a great time to living in this IT era! Following this trend, and due to some need I had to start to learn
a new language, GO.

Go, also called golang, was developed at Google. The motivation was to have a language less complex than C++.

Go has great advantages, such as:

  • Concurrency system, far superior than Python, for instance.
  • Compiled language, which makes it faster
  • The executable is static linked, which allows to create an executable for the destination platform. It means, it doesn’t need an interpreter.
  • strongly handling exceptions
  • Low memory footprint
  • Very easy to learn
  • Docker is built in Go

Starting

Installation

Just visit the Golang web site
The binaries are here.

Baby steps

First things first: IDEs

Learning

I started with the “official” tutorial A Tour of Go

The big points:

  • Every Go program is made up of packages
  • Programs start running in package main
  • By convention, the package name is the same as the last element of the import path
  • In Go, a name is exported if it begins with a capital letter
  • In functions, the type comes after the variable name
  • When two or more consecutive named function parameters share a type, you can omit the type from all but the last
  • A function can return any number of results
  • Go’s return values may be named. Also called naked return
  • The var statement declares a list of variables (package or function level)
  • Inside a function, the := short assignment statement can be used in place of a var declaration with implicit type
  • The expression T(v) converts the value v to the type T (casting)
  • Constants are declared with the const keyword
  • loop: for – like c, java, c++ but without the parentesis
  • The while> in Go its the for. for {} = infine loop
  • if doesnt need the parentesis
  • the if> allows a statement before the condition: if v := math.Pow(x, n); condition {
  • switch doesnt have the “break”, it is implicit
  • switch without a condition is the same as switch true. Good for long sequence of “if”
  • A defer statement defers the execution of a function until the surrounding function returns. defer add(2,3) PS: call’s arguments are evaluated immediately
  • Go has pointers!
  • The type *T is a pointer to a T value. Its zero value is nil.
  • The &var operator generates a pointer to its operand.
  • Unlike C, Go has no pointer arithmetic.
  • A struct is a collection of fields “type Complex struct”
  • arrays: var a [10]int
  • slices: var s []int = primes[1:4] – A slice does not store any data, it just describes a section of an underlying array
  • A slice has both a length (number of elements it contains) and a capacity (the number of elements in the underlying array)
  • Slices of slices
  • A map maps keys to values: var m map[string]int
  • Go functions may be closures. return func(x int) int { return x }
  • Go does not have classes. However, you can define methods on types. func (v Vertex) Abs() float64 {}
  • An interface type is defined as a set of method signatures. type Abser interface {Abs() float64}
  • Interfaces are implemented implicitly
  • The empty interface – may hold values of any type
  • A type assertion provides access to an interface value’s underlying concrete value. t := i.(T)
  • ype switches switch v := i.(type) { case T:…
  • Go programs express error state with error values. type error interface {…
  • The io package specifies the io.Reader interface.
  • A goroutine is a lightweight thread managed by the Go runtime. go f(x, y, z)
  • Channels are a typed conduit through which you can send and receive values with the channel operator, <-.
  • Like maps and slices, channels must be created before use: ch :=make(chan int)
  • Channels can be buffered.
  • The select statement lets a goroutine wait on multiple communication operations.

Conclusion

In a first impression I would say its a very good language for scripting. No complications around object oriented concepts. Just got the best of native languages and modern languages. While reading some articles everyone has the same option, Go its very
suited for concurrent applications, since its very easy to use channels, select and goroutines.
For sure, a very good challenger to my usual preference on python

dynamic and ExpandoObject

tree_volteretaThe dynamic and ExpandoObject have a relation of love and hate with .NET developers. One side defends strongly typed objects to handle data, so it’s possible to handle a lot of issues at compile time. The other side, are developers used to develop scripts, python, and in other similar languages. They defend the dynamic nature of the current projects, with a lot of Xml, Json and unstructured data, so they prefer to use the more dynamic and functional structures of C#. Both are right! Some like to live strong typed and predictable, others like to live on the edge!

Personally, I am a hybrid developer, I like the strong type of languages, but love python and dynamic freedom, like C# provides. So it’s usual for me to use strongly typed objects on my projects, but every time I have to deal with unstructured data, I use the dynamic word and other constructs like Tuple class.

Because sometimes there are too many DTOs on the project!

ExpandoObject Basic Usage

ExpandoObject with methods

Reflection on the ExpandoObject

Why non-requirements are “more” important than requirements

quality attributesWhen I was studying more deeply software architectures, I learned about the most important thing to consider in an architecture (after stakeholders), the non-functional requirements. Yes! Not the functional requirements, but the non-functional requirements.
Before I explain this more careful, let me say what “I” call requirements and non-requirements. (I will use requirements from now on, meaning functional requirements and the same applies for non-requirements)
Requirements are the features the final customer wants. Something like, I want an intranet portal where all the enterprise information is presented, where the users can consult news about the company and where the users can request material, etc. Preference, very detailed, the project manager and all the developers thank you for that.
Non-requirements are the “features”/attributes not related with the business, but very important, like performance, reliability, modifiability, security, auditing, transactional, availability, interoperability, testability, usability and others.
Sometimes people mix requirements with non-requirements, and sometimes they are right. If I want to build a race car, probably performance is a requirement.
The non-requirements are also called quality attributes because they give quality to the architecture. You can build an ugly/raw intranet portal with all the requirements in there. You will have all the information and all the requirements you requested to be implemented, but as soon you present the final product to the end users you will find why the non-requirements are so important. Probably you will call soon a designer to bring some quality to the web pages, Usability. Then you will start to hear some complaints about the performance, a user opens a page and all that information takes a long time to appear. It is time to for the non-requirement, Performance. The users they need important information available on the portal, to do their work along the day, so you will want Availability. The portal should access information available in other systems, and then again, Interoperability, and we can go on and on about this.
Why is this important, and really important? These non-requirements or quality attributes change the way something is built. They should be considered before the development, for many reasons: technical, time, resources and financial, to say at least. Technical, because the architect should consider them in the design and development process. Time, because it will take more time to implement them that just code the raw functionality. Resources, because you will need to consider people expert in some areas, like designers, integration, etc. Financial, as a consequence of the others.
For instance, for usability you should consider experts in that area, like designers. For security, one of the most intrusive quality attributes, you probably need an expert in that difficult area, or consider what blocks of code or modules should be protected and how. It can be a demanding task. For interoperability, you will have to consider what information to consume and provide, protocols, security, etc. If you work in financial software, you will surely want to test deeply the software, so the code and the architecture should be testable, the code will be organized for testability, and a lot of extra code will be implemented for testing purposes.
When you buy a house, the attributes you will take into your decision are more about quality than the living ones. If you follow only the living ones, any space with some blocks adjacent to each other will be enough for living! But, you will consider the access (security), how easy is to move around the house (usability), how strong are the house, like walls, material, etc. (reliability), and if the house is custom build for you, most of the time you will be speaking to the architect to consider this and that quality attribute, you will not be telling him that the house if just for living there, sleep, being and eat. You will want more, you will want quality, you will want a house based, above all, in non-requirements, responsible for making your expected requirements richer.
Curiously, the acceptance of any project, will be based on how the requirements were implemented served by the richness brought by the non-requirements. At least if you want a customer happy.
Without the non-requirements, the requirements will, paradoxically, not be “acceptable” by the final customer.

Architecture – Its all about stakeholders – Part III

jigsaw

In the part I of these series it was introduced the importance of the stakeholders in the architecture. In Part II we have learned how to communicate with them, using views and viewpoints. We will now focus on the concept of perspectives, the non-functional requirements in our architecture but usually the most important, like security, audit, logging.

Perspectives

In the previous parts, we have talked about architecture elements, and how to show them to stakeholders in a way understandable for all the parties. By now, we have an initial structure of the whole system, but it misses the most important things in the architecture, and usually the more challenge to include, the quality attributes, or sometimes called cross-cutting concerns, like security, performance, logging, audit, scalability, etc.
These attributes of the architecture are always very challenge to include because they are orthogonal in the architecture. For instance, logging is something we want all over the architecture, probably in all modules, tiers, layers, etc, which implies changing all views already developed.
The name was given to these attributes in the context of an architecture its a little bit controversial, because there are entities that consider these attributes additional views of the architecture, and other entities consider something that change the views, orthogonal in the views, something that complements and can change an entire viewpoint. So it is natural to find these cross-cutting concerns named as views or perspectives. I, personally, like the term perspective, because these attributes are not another kind of views in the architecture, but the application of “quality” in the architecture, the insertion of elements that enrich the system. Like in a house, besides the construction we can improve the whole house with a better light system, internet all over the house, better walls, toilets with material that don’t get rusty, etc.
Why is this important for stakeholders and why we need them? Because they are usually the most important concerns! For instance, in a Bank, security is probably the attribute most important, and I bet almost every requirement includes security, explicitly or implicitly. The people who give support surely they want logging and audit to help them finding the origin of issues or someone who tried to access a resource illegally. In a website like amazon, the end users surely they want a very performant website, and so amazon wants a scalable system, for the system to grow big and in a smooth way.
The quality requirements, are the core of the requirements. They give color and form to the functional requirements. You can create a web page to register a user, but you will want to include audit, logging, security (like HTTPS) and manage the load balancing and scalability, not forgetting about sticky sessions or another architecture tactic (different from patterns).
With all these views, viewpoints and perspectives, you can think the architect will have a fragmented view of the whole architecture, which is true, but someone who wants to be an architect must be prepared for this challenge, find the best way to build a system with all these pieces, that’s why he/she will be an architect. It is not about the technical skills, only, but about choices and stakeholders.

Architecture – Its all about Stakeholders – Part II

<">viewpoints views

In the previous post, we have seen the importance of the stakeholders in the development of an architecture. Because stakeholders aren’t all fluent in technical language we must find a common ground to communicate.

Views and Viewpoints

An architecture is a very complex system. If you try to create one big diagram to show all the elements of the architecture and its relations, it will be a big document, only useful for the ego boost of the architect. The stakeholders don’t need do know everything about the architecture, just the parts with interest to them. For instance, users of a bank application, they are more concerned with the user interface, performance, available information and in many cases with security. They don’t care how the database server exists or how is it connected with web services, integration tiers, etc. The architect should create a set of views of the system, for each set of concerns of the stakeholders. To this set of views, we call it Viewpoint, a part of the architecture. One view always belongs to a Viewpoint, and a Viewpoint can have multiple views. Examples of viewpoints are information, development, concurrency, functional, deployment and so on. We can observe there is a set of stakeholders with an interest in each of these viewpoints.

Let’s see an example. When building a house, you don’t want a unique blueprint with every element of the house, the exterior, the interior, the materials, the flow of the light, water, electricity, streets around, halls, etc. You expect to have several views of the house. You will have some views for the exterior like front view, side view on the afternoon, or if you want to see how the interior will look, you have a view of the living room, another for the kitchen, rooms, toilets, and so on. Other concerns could be how you circulate inside the house, and you expect to see views from the entrance, hall, the accesses between rooms and living room and toilets, etc.
The views about the accesses are a viewpoint related to the circulation of people inside the house. The other views are more related to the context, how the house is presented from outside, for instance.
The house is the same, but the views are different, and directed to specific concerns and different persons (stakeholders). When the architect talks to you, shows how the house will be at the end, but when he talks to the company that builds the house, the concerns will be completely different, they are interested in exact measures, materials, ground, time. These diagrams will have information you don’t care because they are too technical.

Again, the stakeholders are the fundamental piece of the architecture, besides the project being developed for them, even the management activities are around them and depend on them. Your strategy as an architect should have the stakeholders as the main actor on the scene, and all the tactics should include them in every decision.

Now, we know how to talk with stakeholders during the development process of the architecture, it is time to include in these views quality attributes, or sometimes called cross-cutting concerns, like security, performance, cache, etc. We call them perspectives (it is not a consensual term in the architecture community, but the main idea exists and is shared). We will see more about this in Part III of this article.

Architecture – Its all about Stakeholders – Part I

stakeholders

In this three parts article, I will introduce the concept of how the success of an architecture depends on the relation with the Stakeholders. Part I will describe the influence of Stakeholders in the success of an architecture, then Part II e III will describe how to communicate and create a common ground between the architect and the Stakeholders.

Stakeholders

A Stakeholder is: “individual, team, organization, or classes thereof, having an interest in a system” (ISO/IEC 42010).

From the definition, we can observe something all Stakeholders have”…interest in a system”. This interest can be from a perspective of the customer who is buying the system and follows closely all the steps in the creation of the product, until someone, anonymous, who just uses it, like a person or system buying a ticket to a game. So, the objective of an architect should be building a system maximizing the satisfaction of all Stakeholders.

An architecture is, therefore, built around the Stakeholders needs. The majority of the Stakeholders are people, so the human variable enters in scene very strongly in the construction of the system. Things like negotiation, agreements, meetings, communication, expectations management, are skills the architect should dominate.

An architect is not a Project Manager but should work very closely with him. The success of a project from the Project Manager perspective is the satisfaction of the Stakeholders, as so for the architect. That’s the reason, for some many times we see the architect as a project manager too.

The process of developing an architecture evolves thinking, technical skills and a lot of communication with Stakeholders. The Stakeholders, usually are persons who don’t understand technicalities. It is necessary, for the architect, to create a way to expose his/her views to them, to create a common language of understanding. For that, we need Views and ViewPoints, something I will talk about on the Part II of this article.

Being a better architect – Developing for several markets

apple_microsoft_android

An architect should always be ready to increase is knowledge considering the actual market tendencies, because it is his job to present solutions to his customers, which are always considering solutions to decrease their costs using IT. With this vision, as an architect I am always pushing myself with new challenges and learning more and more. Since the beginning of the current year I have bought books, in subjects like IT, social behavior, strategy and soccer. I like to develop myself in different subjects in order to acquire other perspectives and expand my way of thinking. I believe it increases my creativity and give me thinking tools to handle challenging projects.

Two weeks ago, I decided to develop an application and publish it in different App Stores, I want to know the software production lifecycle in all these markets:

  • Android Play Store
  • Apple Store
  • Google Chrome Extension
  • Firefox Add On
  • Windows Metro
  • Windows Phone
  • (Maybe) Android Watch
  • (Maybe) Apple iWatch

The idea was to learn how these markets/stores work. What is to develop an application in different languages, different technologies and different paradigms.

After some analysis, I decided not to use popular developing frameworks, where your code once and they will produce code for the several markets. I wanted to use the native environment/language, Developing using HTML5/JavaScript is about design, not about acquiring some know how in each environment. I wanted to use the knowledge I already had in languages like Java, C#, JavaScript and HTML, XAML and little bit of Objective C, and maybe learn the new Swift Apple language.

At the end, my expectations are to be happy with the result. Knowing I have learned how to develop to the most important markets and be a better technical team leader, architect and have increased my know how to manage these kind of projects.

The process

I already started with this professional project/program. I have used a three step approach:

First, what kind of application to develop.

I have a lot of ideas for apps, so I chose one where I could develop more than one screen, should have some utility and a basis for increments and upgrades. For version 1.0, I just wanted to release it, not earning money or using traffic or Ads, just be in the market to get the know how about planning, developing and publishing it.

 

Second, decide the target markets/platforms.

I chose the six platforms/markets above, I believed they were the big ones.

Chrome and Firefox extensions are in the list to improve the knowledge in the browser arena, I think it is important for an architect having a little bit of knowledge in the user applications/tools we use daily. It helps understand the end user needs.

The “watch” items, lets see if I have time and energy. The issue with developing for several markets, it’s the management, not the developing.

 

Third, the order of development.

I started with the Chrome Extension, after researching how to develop one. It is simple, just HTML, JavaScript, and a manifest. Some screenshots and marketing images, and it is done. A good basis of know how for the future developments.

 

Next steps

I am now developing the windows version for Windows Phone and Windows Metro. I have already developed the Android App and Chrome Extension. You can follow all these apps on my blog at “Give Me Numbers – Apps”.

In future posts I will describe the process I have followed for each app. Including, at the end, a comparison between all technologies from the perspective of a developer/publisher and manager

Hadoop in 5 minutes for beginners

01_Hadoop_full

So, you don’t know nothing about Hadoop and want to have just a simple picture of it? This post its for you!

So, you have a lot of data (TBs or more), spread all over the place sometimes structured sometimes not structured and you want to query these data. You are thinking by now, I will need a lot of power to query data “organized” like this. Yes, you need, you need Hadoop and all the Big Data techs around it.

What is Hadoop?

Wikipedia has this interesting fact: “Hadoop was created by Doug Cutting and Mike Cafarella in 2005. Cutting, who was working at Yahoo! at the time,[6] named it after his son’s toy elephant

 

As Apache states “The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.”

What?

You have a system (that can be logically organized in  cluster >> racks >> nodes) with several components to handle distributed processing and files between a lot of machines. You have, between others, HDFS, a distributed file system, and an implementation of the Map Reduce pattern.

HDFS is a file system that works over all machines in the system, but you only see it as one file system, because is is distributed over several machines. How about my local file system? Still exists, the HDFS works over your local file system. (ex: “hadoop fs –ls“ a command in your local file system to runs a “ls” in the HDFS)

The MapReduce is a pattern (see here) to process large data sets (ok, you can use for small data sets too, because it’s a pattern, not a product, and you can implement it in any language you want with very little code). Hadoop uses this pattern to run your queries over the data. (It uses tasks, jobs, etc. for your requests, but always using this pattern in the execution)

So, by now you have a Distributed File System and an engine of tasks and jobs to run applications implemented using the Map Reduce pattern. Yes you are right.

So, how can I query all this data? Well, you can implement applications in any language, usually Java where you control the tasks, jobs, the Map and Reduce functions for the Map Reduce pattern, etc. A lot of work to do. Well, you can use other techs of Big Data that will help you to implement these queries and handle operations over your data, these are some of the languages(platforms) you can use and simplify your programmers life:

PIG (yes, it is this name) – example extracted from Apache. Loading and saving data…

Hive (“The Apache Hive™ data warehouse software facilitates querying and managing large datasets residing in distributed storage.”) – some examples from Apache? More SQL like but it is not SQL as we know it.

Jaql – A JSON Language used now by IBM BigInsights. An example from the Project Page

//
// Bind to variable
$log  = read(hdfs(“log”));
$user = read(hdfs(“user”));

//
// Query 1: filter and transform
$log
-> filter $.from == 101
-> transform { mandatory: $.msg };

// result …
[
{
“mandatory”: “Hello, world!”
}
]

Others –The are other projects and languages but for a introduction I think these 3 shows different ways of querying the data.

 

The overall picture

You install Hadoop, you will have an HDFS and Map Reduce engine. For querying the data you can develop yourself code or you can use languages like (PIG, HIVE, JAQL, …) to handle all the Map Reduce stuff behind the scenes. Yes, all the querying from these languages are always translated to tasks which run Map Reduce patterns, you don’t have to worry about the Map Reduce implementation, that is why its fast and your processing and data could be spread over thousands of machines!

The 10 Types of Testing you should know.

testIf you aspire to be or you already are a Senior in Software Industry you must understand the language used in all the development cycles of the project, and testing its one of the most important, because its about quality to the end user (at least). Besides this, sometimes its necessary to develop the project thinking about the kind of tests that will be done in the product, for instance, if there will be an automated test in your product maybe you have to prepare the architecture and code to facilitate the implementation of this kind of test (without compromising the core behavior of the application). One example of this is the use of the inversion of control and dependency injection patterns, they  facilitate a lot the using of Mocks, which are very used in unit testing and other types of testing. So now you have a software decision in the architecture because of the way you will want to test your application.

 

Smoke Testing

Usually the first tests after each release before the application goes to the “real” testing. It tests major behavior of the application to see if it is minimally stable before it can go to regression and functional testing. Just tests major functionality, like open windows, see if buttons are working, grids are displaying results. It tries to detect major problems in the application before going to test.

The name Smoke Test comes form the way some hardware was tested, you switch on the hardware and observe if there is smoke arising. Other examples are from plumbing, where artificially smoke is injected in the pipes to see if there are leaks.

Functional Testing

As the name says the objective is to test the functionality of the application against the functional specification document. One of the most important tests because it tests the purpose/conformity of the application.

Non Functional Testing

If functional testing tests the functional requirements, the non functional testing tests the non functional requirements, like performance, compatibilities, ergonomics, security, stress, reliability, scalability, etc. Depending of the application, these tests are so important as the functional tests. If you consider a commercial web site, it can have a great design and almost 100% bug free, but if it is slow for the end user is a question of time until no one visit it anymore.

Regression Testing

This test is used to ensure that changes made to the application in fixes or enhancements are not impacting the functionality that was already working. Usually the execution of unit tests are included here.

Sanity Testing

There are a lot of people thinking that this test it’s the same as Smoke Tests. But are subtle differences between them:

Smoke Tests

  • Usually scripted
  • Ensure that the major and crucial functions of the application are working
  • Applied before going to functional and regression testing

Sanity Tests

  • It’s a kind of small regression test narrowed to a small set of functional areas
  • Usually is unscripted
  • Is used to find potential problems after changes in a specific area of the application

Similar to Smoke Tests but the main idea is to test only the changes, not run all the major tests.

Exploratory Testing

This test is sometimes not understood by many people because the idea is to explore the product. In this test, the tester will use his knowledge and experience to explore the product with the objective to find problems, develop new tests, to know better the product, etc., just like the name indicates, exploring.

This test is conducted when the tester doesn’t have yet the requirements document, the time available to test is short or the application is still in early stages of development.

Usability Testing

“It takes only five users to uncover 80 percent of high-level usability problems” Jakob Nielsen

This is a black-box testing technique where the aim is to observe end users using the product to discover errors or areas of improvement. You can see this kind of testing in beta products deployed to end users and even in final products where you find an option to send information (usually anonymous) regarding your behavior using the product.

System Testing

This is the test of the product as a whole. This test tests the functional requirements but at the end user perspective, where the functionality and the architecture are included, because you are testing the whole product. This tests is executed in an environment similar to the production environment.

User Acceptance Testing

This is the test to be done, before the application is accepted and go to live production. Usually is realized after the system testing. Sometimes is called Customer Acceptance Testing.

Installation Testing

This test its the final one, where the process of the installation of the software is analyzed, considering space in the storages, permissions and other factors related with the production environment.

 

Final thoughts

There are a lot to talk about each of these types of testing and if you want to know more you can find great sites and books with lots of information about it. I think that a superficial knowledge of all this kind of tests is a requirement to any project manager, team leader or senior developer, so they can know how to talk to the test teams and to the customer (ex:User Acceptance Testing). A junior developer should know too some of them, at least, regression testing for instance.

As a developer you probably are questioning, “what about unit testing?”. Unit tests are developed by the developers to ensure the quality of internal code, and usually are integrated in Regression Testing.

Bad Behavior has blocked 154 access attempts in the last 7 days.

Hyper Smash