Sunday, August 19, 2012

Dealing with Criticism

0 comments
This post is about tips of how to criticize, raise concerns, and how to deal with critics in the software industry.

One thing I've learned over the years working in software development is that, no matter what one does, there will be criticism about your work. The more visible and high profile the work is, the more it will be criticized, no matter what. Not being target of criticism would mean very simply that it satisfies all people involved, which is almost never the case. The reason for that is that there are always different approaches to the problem, different approaches on how to document and/or present the problem and solution, among others. A person understanding your work could be highly logical and prefer a very logical/mathematical approach, another can prefer a higher-level approach, while still another person would have liked to see a more business oriented perspective, and a few more would have wanted more technical low level details than you presented. I personally haven't found a way to satisfy all type of audience - if it is at all possible, it will extremely hard and time-consuming and very hard to find this investment worth the time.

Given that, no wonder that criticism will rise among people that get in touch with your work. For very personal (and sometimes petty) reasons or for good business reasons, there will be criticism. In the software industry, in particular, this is bound to happen frequently as engineers are smart and opinionated individuals. The issue becomes more severe as we're in a highly creative field where there is typically no single solution to the problem at hand.

Being Critical

There are many reasons to be critical, and for everyone interested in growing as a person and professional, this is an area that provides tremendous opportunity of growth if you have the right intent and learn how to provide feedback in the right way.

The most important lesson is: first, before being critical, try to think that there could have been reasons to make a decision in that why. Or if that's not the case, it just means that the person does not have your experience and knowledge, which is a great opportunity for your to influence that work.

Author does not have your experience or knowledge

One thing is certain: no two people have the same experience or knowledge. On a particular topic, it's possible that one has more expertise or knowledge, but overall there is always something to be learned from other.

In the case the author of a work does not have your experience on that topic, this is a tremendous opportunity for you to teach and help a fellow coworker. The goal is not to make your coworker feel bad, but actually walk him/her through what you've learned and see the problem through his/her own eyes. It's not about forcing your suggestion on how to do things, but being patient to get the person to see the potential problem you see and potential consequences. If the final solution addresses the problems or consequences, that is what matters.

For the inexperienced author, if your experience indeed applies to the problem, either the person will see or will take the risk of running into the same problem. If the latter, there are two courses of action. First, if it has a great potential to impact the business and the person refuses to see, it may be worth involving others to see if through consensus the argument becomes more compelling . In some other cases, though, the best course of action is to let the person move ahead and take the risk. Either he/she will learn from it in the future, or something is or will be different than your own past experience and it may not be that bad or impactful to the business as you had pictured. Either way, both will learn from the experience and the organization can benefit from this as it will mature.

There was a reason for the decision

Another possibility - often not considered - is that you could be criticizing something without understanding the reason a decision was made that way. Actually, not only the reason, but you may not the entire thought process or context in mind. With that, the author may have considered your point and, among other options and pros/cons, it could have been still the best alternative. This is where you distinguish mature engineers (and very often great engineers from the merely good ones): you are willing to be humble at first and potentially learn from others. As times goes by, these engineers are increasingly right most of the time, but still there are points that these engineers can learn from and fill the gaps in their knowledge and experience - and all of that by just trying to understand others' thought process.

Biases when evaluating someone else's work

As mentioned above, you can have two biases when taking a look at someone else's work: to think that the author might be more intelligent or more knowledgeable than you or that the author is stupid and did not see all of your points. Any of these extremes can be a problem on how you will criticize someone's work.

Thinking that the author is always more experienced, more knowledgeable or more intelligent can hold you back - you may have good suggestions or good criticisms, but this belief makes you not express yourself and be vocal. On the plus side, though, it may make you learn from every single experience if you know to ask the right questions and understand how the author got to that conclusion.

On the other hand, though, thinking that someone is always less knowledgeable, less experienced, or less intelligent can be a trap. This is very common in companies where people are more aggressive to get things done. In this case, you can end up pushing your view on others, even though they may have valid points and alternatives. Also, it can make you less open to the different perspectives, reducing your possibilities of learning something. Long term, the consequences are that you tend to keep doing things in the same way, less likely to adapt to changes or improve processes.

Ideally, you can hit a sweep spot in the middle of these extremes. First and foremost, always consider the possibility that there was a good reason for someone to be done in a given way. And consider your own experience and concerns as potentially valid.

Find what concerns you, not how you'd do it

Problems may have a number of different solutions - in the software industry, this is even truer as there can be so many potential combinations of solutions when dig into the solutions. When tackling a problem, you will probably have a different approach than someone else. It may be similar, but it won't probably exactly the same.

Given that there are multiple solutions, the first thing is to respect the creative process that someone else went through. The solution may not be the same as yours, but it can have merits and can take into account more than you're able to see. So do not focus on HOW you would do it, but rather on what IS your concern and what are the undesirable consequences.

Be respectful of other's time and efforts

If you have too many concerns and disagreements, in order to be mindful of your time and your colleague's, and also that this is a business environment where you must be efficient, have FOCUS and priorities. If you have many points to get across, prioritize and pick your battles. Put yourself in the other's shoes and make his/her life easier - while still raising what is important to you.

Get your point across

These are good actions that can be done to perform criticism in practice:
  • Before criticizing, inquiry the author why a decision was made in a certain way
    • This can prove to be very revealing and may allow you to understand the context. Your criticism may not make sense anymore after that.
  • If previous step did not help, first check if the author considered certain scenarios or concerns
    • At that point the author may recall and address your concerns
  • If your concern was not considered, walk the author through
    • A "this is crap" phrase is not the best way to achieve the desired result, but rather try to make the author see what you're seeing. If required, walk him/her through each step of the thought process, how to get that scenario or how that can have an undesirable consequence

At times not even following these steps will actually help you get your point across as not everybody is open or will follow your arguments. In such cases, it may require more data and other measures to be taken offline. In other situations, it can take a long period of time to change the situation as people need to see a pattern of concerns that become reality before starting to give enough attention to you. In general, though, the steps above can help you get your point across in a respectful way, contributing to a healthier work environment.

Learning How to Deal with Criticism

The flip side of the coin is how to deal with criticism. This can be very tough, as not necessarily all critics will be acting out of good intentions. Even if they do, they may not have spent the amount of time you did thinking about a problem. And to make it worse, even if they do, they could simply not have seen all the issues and problems. Finally, there are those who are very loud and can make you look bad, which requires some damage control to be exercised.

Know why you're doing what you're doing

Before your work is shown to a wider audience, make sure that you have it reviewed in smaller circles to get the most important questions answered. These small circles must composed of people that you provide valuable feedback, but are also easy to work with.

Moreover, to make the entire process more efficient, make sure you know why you're doing that work, so you don't have questions later that will throw you off track. For this, go beyond your role and understand business requirements, technical requirements, try to understand how it fits in the overall bit picture. That can simplify your life down the road when questions that are related (although not directly applicable) to your work are raised.

Assume good intention

Criticism may arise for a number of reasons, but the healthiest thing to do is to assume all of them have the best of the intentions. If you assume they are not out of good intentions, you may not get it right (how can you say for sure the reason for the criticism?) and rather than focusing on the problem at hand, you will detract your thought process to think on the reasons, and why this is happening, why this is happening to you, etc. Forget the reasons and make the healthy assumption that there's a good intent behind the criticism - that will allow you to focus on the criticism.

Accept the fact: there will be critics

I'd say that accepting that there will be critics is one of the most important things in dealing with criticism. The likelihood just increases as you're doing things that impact more people or are more visible. It's unlikely that you will have criticism if you are doing something very local that nobody cares much, but it's very likely that you will do if you are changing how the Windows or Mac interfaces look like. No matter what, there will be critics, it is a matter of life no matter what you do. Not accepting this fact means that, with every criticism, you will reiterate the debate in your mind that there should be no critics of your work and your work environment (or world) is not fair.

Pick which ones to address

Another point is what criticisms to address - depending on the project and impact, it is completely unfeasible to try to convince all critics of your decisions. It is a necessity to be able to pick which critics to focus on and spend your time on these critics and their points. This shows another sign of maturity and something one learns as time goes by. In some cases, there will be valid criticisms that can and should be addressed. If many people are involved and they all have different opinions, it can be just unfeasible to address all of them, so decide the criteria to follow, like: technical/business expertise or hierarchy with the company.

Depending on when the criticisms take place in the process, my suggestion is to make sure to pay some attention to those who disagree with you, as they may either point valid concerns or will help you prepare better for further justifications down the line if these criticisms come up again.

Know how to defend your points

Over time, all of us forget the reasons decisions were made. We may not even remember the decision later on, and if we do, we tend to forget what led us to that. From that point on, it is very hard to be able to defend your decision as either you will need to go through the entire thought process or simply will not remember.

For future questioning and also for you to organize your thoughts, write down the reason decisions were made, from all possible perspectives. Make these answers public, if you like, or write them in a private document you can go back to if you need in the future.

In addition to that, communicate them in an intelligible and organized way, adapting the answers to each audience. Writing down your decisions and reasons is one way to help you organize your thoughts beforehand, but if you have a different technique, make use of it. Know how to communicate them to the right level of detail and focus depending on the audience: for instance, a business audience has a different focus than a technical audience and isn't commonly interested in low level details.

There will be other ways to do the same

Many critics don't think what bothers, but focus on how they would do it and that is what is thrown at you as a criticism. This is a trap, so don't fall for it. If you do, you will end up trying to point out what is the problem with how they would do it, rather than focusing on your solution (and its problems). Make sure to ask for the clear concerns on your solution and if the critic can't state that, follow up at some other time, but don't fall for the "why don't you do it like this?" question.

Be mindful of your own time

Make sure to get your critics to focus on the important concerns, especially if you have many critics or many points to address. In many occasions it's impractical to answer all questions from all people, so it's your job to know how to request feedback and not prolong discussions that don't offer much value. Sometimes the best thing is to capture these critics in emails or documents and be transparent upfront that they will be addressed if time allows - this gives you the opportunity to do in the future, but not to hold your work until every single discussion on why you used this notation vs. that notation is sorted out.

Finally, another suggestion is to know what the agenda for emails or meetings are and know how to get back on track when discussions are sidetracked due to unfounded criticisms.

--

I hope these tips can help you deal with this vital and sometimes painful process of criticizing or being criticized. It is definitely more an art than a science and there are not hard rules here.

Additional tips on how to deal with criticism are very welcome. Please share if you had different experiences with critics.

Saturday, August 18, 2012

Make: Electronics - Experiment 18 (Part 2)

0 comments
This is the part 2 of Experiment 18. It's a "reaction tester". By using 555 timers, there's a delay start to a "counter". First one presses a switch and a few seconds later, the LED goes off and it starts counter. The game is that, whenever the LED goes off, the user presses the button as quickly as he/she can. The numeric display shows how long in milliseconds (not necessarily very precise, though) to react.

Wednesday, August 15, 2012

Make: Electronics - Experiment 18 (Part 1)

0 comments
This is an experiment with 3 4026 counter chips and a 3-digit LED display.

Eventual consistent systems using ACID as building block

0 comments
I came across the following post, which is very interesting. It talks about how we can have subsystems being ACID, but the global system being eventual consistent. The analogy is founded in real systems and thinking in these terms can be a good tool to help you seemingly complex or impossible problems.

ACID as a basic building block of eventually consistent, distributed transactions

Monday, August 13, 2012

HBase and HDFS replication

0 comments
As I learned about HBase and HDFS, I wanted to understand how HDFS actually does its replication, whether it's an synchronous replication, what is the flow. As it turns out, it wasn't easy to find the answers to these questions, but I ran into a very good page on the internals and how HDFS works that is worth sharing:

Understanding Hadoop Clusters and the Network

Another aspect of HBase and HDFS is the ingenuity of having a filesystem that is good enough for Hadoop and then building a database on top of it, without reinventing the wheel. Kudos to Google for their work on GFS, MapReduce, and BigTable, which inspired the open source implementations.

Sunday, August 12, 2012

Working at big software companies

12 comments
For nearly 7 years I've worked at 3 big software companies - all of them with tens of thousands of employees. I also had 5+ years working in research environment, at startups, consultant, or by myself, what gives me a more complete perspective to look at big companies.

Throughout the time I've worked at big companies, I had many philosophical conflicts between what I was experiencing and what I felt was right for myself. In this post I reflect on these conflicts and the my current stand on big companies.

Not to give much away upfront, but overall making sense of what working at big companies means is an extremely fascinating experience at the end of the day. Although oftentimes this experience may not align with your personal values and how you want to spend your life, it's definitely quite challenging and provides lot of personal growth opportunities. On the other hand, even if not evolving at the pace you might want, there are lot of hard/soft skills that can be learned at these companies.

Enough said. Let's start with the positive aspects and then get to the more contentious ones.

Powerful

One obvious thing of working at big companies is how powerful they are. If your project is important to the business in some way, funding will be guaranteed. If your project doesn't work out, in the software field you will most definitely be reallocated and there will be no risk of losing your job. Especially in an area where there's deficit of good talent, rarely a company can afford to lose employees. That is just really amazing.

Not only funding, but given the big army of employees and global reach of these companies, it is amazing how much impact your work can have. You may not really have a lot of input to guide some decisions, but there's no question that depending on what you do, there will be a big reach.

Scale

Related to the topic below, if you're in the technology business, more often than not the opportunities of working with big systems (like big customer base, big data) occurs at big companies. More recently even some startups are reaching this scale at times, but still big companies tend to be a best bet in that sense.

Other aspects of the business, like having very big build systems, or having well defined and matured processes for support will be only found at big companies, so working at them turns out to be a great opportunity to learn from systems that evolved over time.

Perks

Although there are differences among companies, big companies in general have at least some standard benefits that are hard to beat - and small companies generally can't compete until they get to a certain size.

Career

Due to their scale, there isn't one person or small group overseeing every aspect of the business at a big company. There is a need to scale out and delegate power to others to make decisions, to decide priorities, and to help employees grow into their careers. For this not to be chaotic and to reduce politics among these people in charge, what do we have at big companies? Processes. Just like a government, we do have regulation determining what can/should be done or not in order to allow the company to scale out in a relatively organized and predictable fashion.

Career progression is particularly interesting as what happens is that, irrespective of how this is implemented, companies need a model and process that replaces underperformers and reward top performers in some way. Another goal is that all must end up having the feeling that they are progressing, so there is a model for career progress (i.e. promotion). This model must work in general for the majority of the employees - they must good enough for a substantial part of the employees, and definitely will not work for 100% (I haven't seen or heard of any that is even close to that). Attrition rate (i.e. how many are leaving the company) is tracked and adjustments are put in place when the company realizes the model is not working well. Sometimes changes are put in place because another big company found a model that is better in some sense and is attracting more people.

Top Performers

For top performers, as mentioned, they must be rewarded. They tend to go through the company ranks - step by step (or more precisely level by level). Why step by step? There is a process in place and all must abide by it. It doesn't matter if the employee is absolutely fantastic, but there is a limit of how much and how frequent the company can actually reward that employee. The only exception is if there is a strong business need that makes the company circumvent the process, but that involves a lot of effort by a set of managers and creates a precedent that can harm the model for all other employees. As you can imagine then, typically that is not the case and I've seen these being "one in a million" cases. Leaving these exceptional cases aside, typically top performers end up being split into two broad categories:
  • Patient top performers focus on the job and are fine progressing as the company's career ladder allows
  • Impatient top performers do not accept to be told how fast they can progress, and end up leaving for a different job or career
Of course these categories are just very high level. One thing that determines how "patient" a top performer will be is the types of assignments he will have and the changes that he will be progressing from his perspective (not from the title/position perspective). If many circumstances are favorable (business priorities align in a way, his/her manager views the potential, etc.), some people stick long enough as they have the feeling of progression irrespective of how fast the company's career model allows.

Holding Back

In many cases, though, these top performers will be held back in favor of a "greater good", as the overall team morale must be considered. That way, other people, even if not top performers, must be happy and they will be taken into account for task assignments, etc. Sometimes some compromises will be made in order to keep the team "stable", so the business can keep going. Many variables are at play, and what is important here is that top performers may be "sacrificed" by having tasks that may not be as challenging as they can do and will need to have patience. That is just the way it is and the employee gains this maturity over time and learns to deal with if he/she decides to stay around.

Weakest Link

Another aspect of holding back is that, irrespective of how well one does the work, at big companies you have to work with many other people and can't be in isolation. Depending on the organization, you may work closely with professionals from other organizations within your company that you don't have any control of. As mentioned above, no owner is overseeing the business, and that can mean simply that you will be held back by whoever you're working with - be these people your peers or managers. More hierarchical companies have more of this problem, but it actually occurs across the board, and to keep working at a big company means that you just need to comply with that.

What does it mean to a top performer? That he/she needs to slow down and go at the speed of the weakest link. If the weakest link is just one person, management could fix that, but if the organization is the "weakest link" that just functions at a different pace, there's not way around (well - unless you try to change the organization, what can be almost impossible, but could be an interesting personal challenge).

Some times everything will move forward to the contentment of all, other times it will be extremely frustrating. And although a top performer can help improve the organization to be more efficient, note that it is extremely hard to bring everybody to the same level, either due to skills or even to their lack of interest in improving.

You're not working with the owner

Because in big companies you're under a manager, this person is not necessarily vested into the company like an owner, as he/she doesn't have so much at stake - both if the company succeeds and if the company fails. Also this person has his/her own career to take care of, has his own biases, and not necessarily cares about that his/her reports are individually being challenged to their limit. That is not the goal. And the most important point is that having layers of managers mean that not necessarily productivity will be the currency - politics and how your work is portrayed to others may have much more impact on your career than actually how productive you are.

Decision Process

One is farther from the decision process as the organizations are bigger. Even with valid input, the company simply does not scale by listening to everybody and taking everyone's concerns into consideration. Because of that, the company misses great feedback at all levels (high level direction all the way to the high level architecture of your product/service). Unless you're at the level that is listened to, you will be limited in how much you can actually impact until you grow step by step to that point at which you will be listened to. Of course anyone college grad with great opinions since day 1 will be dismissed for the most part - not giving this person's the opportunity of learning and pushing him/herself at the pace he/she could.

Innovation

I like to see this aspect asking this question: what is the process for someone to improve a code or a process at big companies? It varies a lot in my opinion. The worst case I've seen was: for absolutely any improvement, you need to have approval from 3 managers. Yes, even if done on your time, at home on weekends, fixing that behavior or trying to integrate that new interesting framework will not be allowed to be checked into the source code repository unless you convince managers of that. Being typically risk-averse, unless there is a a good business reason, your request will not be granted. Having approval from 3 managers was the worst case, but other big companies also had similar behaviors - and when improvements or innovation was allowed, it was just because your manager was taking the risk and responsibility for any impacts.

Of course what is the implication of all that? The experimentation, that sense of ownership and caring about your code, they all tend to diminish as the processes in place do not encourage that. After all, there's a business to be run, right? What does it actually make to the engineers that come out of college with all that will to change the world? Do they give up little by little?

Even if not thinking on a personal level, I've seen these big companies becoming really good at continuing to do the same as they had been doing before, innovating a little, just what is told by management and supported by the system. Those small innovations that come from almost every single individual with a thinking mind, but mostly by the top performers, are not tapped into. Overtime, though, many of them just learn to comply with the way the big company works. And not only comply, but these people start to think that it is the only and the right way of doing things.

What if big companies are not for you?

Big companies are the perfect place for many different types of people. The rate of people happy at these companies is quite high, so the first thing to do is to evaluate how well you fit within a more rigid organization. If you don't know yet, I definitely recommend having some experience at a big company. Give it some time (one year, at least) and try to learn all the subtle things about it, in order to understand its culture, what the values are, what is encouraged or not. It will be an invaluable experience, even if to learn that it's not the place for you.

If you understand that big companies are not the place for you, then you will need to explore other options. I'd suggest to start with side projects (there's a great blog post about them) and hobbies. They may give you the opportunity to push yourself and learn other things - this could make your life overall much better. You may not want to leave the big company if you're satisfying these needs in some other form.

Another option is to explore startups and perhaps going on your own and founding your company. Side projects could be a good segue to startups or to your own ideas, but even if not, I'd highly recommend to have experience at some startup if after reading this or having your own experience, you feel it is not for you.

Your thoughts?

What are your thoughts on big companies? I'd love to hear them.

Please see comments on Hacker News here.

Saturday, August 11, 2012

How does CNAME resolution work?

0 comments
There are plenty of places on the web explaining the purpose for CNAME DNS Records and how to configure them, so this post is not about aspects of CNAME records, but rather how they are actually resolved.

Among others, some of the questions that can be raised when one thinks about CNAMEs are:
  • Does the client has any knowledge about CNAMEs at all?
  • Do they need to check which records are available and only then perform a resolution specifying the right record type?
  • What are the requests and responses in order to resolve a domain that has a CNAME record rather than an A record?

In order to answer these questions, let's start to answer this question with an example which has a CNAME:
channel9.msdn.com
These are the DNS records currently configured for this domain:
channel9.msdn.com             IN CNAME channel9.trafficmanager.net
channel9.trafficmanager.net   IN CNAME ch9production.cloudapp.net
ch9production.cloudapp.net    IN A     65.52.21.59
Let's fire up the browser, open a traffic sniffer application, and type "channel9.msdn.com". Now let's look at the traffic on the wire below.


As you can see, the query performed is for record type "A". This answers our question on whether the client specifically requests a response for type "CNAME" - the answer is no.

The other point is that the query above has the flag set to be a recursive query. This means that our ISP DNS server will resolve recursively, if required. We will understand recursive in this context in a moment.

Back to our main questions, if our query is on type A, how does the CNAME come into picture then? Let's look at the answer we get from the ISP DNS resolver to help us answer that:


On CNAMEs, we can note that we queried on type A, but received a CNAME record (channel9.trafficmanager.net).

Because of the recursive flag, our ISP DNS resolver went ahead and queried record channel9.trafficmanager.net instead of returning the response. Like our original query, it was type A, and it received another CNAME (ch9production.cloudapp.net). We can try to simulate this query by typing channel9.trafficmanager.net on our browser bar.

Given the recursive flag again, it went ahead and resolved ch9production.cloudapp.net by sending another type A query, which finally resolves an IP for us to connect to. As you can see, in one response we get all the answers for all CNAME resolutions and also the final answer (65.52.21.59).

Answering our other questions:
  • The client does not need to know about whether it's a CNAME or A: it just sends a type A query. If it happens to be a CNAME, it needs to issue a type A query again based on the CNAME response it got.
  • If the recursive flag is set, these requires will be done by the ISP DNS resolver and the client will get all the responses in one shot.

Monday, August 06, 2012

MSTest: Unable to start the agent process

0 comments
When running mstest, I started getting the error:

Failed to queue test run: Unable to start the agent process

There are many reasons to this problem, as one can find on the web. Typically the problem is that a previous process related to MSTest is running and holding some references.

But, as you may guess, this wasn't my case.

MSTest, when it executes, writes a log files called VSTTAgentProcess.log in the same directory as the .exe. I had checked in this .log file, what made it set as read-only. Next time MSTest runs, though, it exits with the error above if unable to write to the log file.

Lessons:
  • If you re having the error above, make sure that you can write to the directory where the .exe is located.
  • If the .log already exists, make sure the file is writable

Intelligent Code Review Tool

0 comments
I had some ideas to improve the code review process...
  • Reviewer suggestion: what if, based on the changed files and lines, the code review tool automatically analyzed all the checkins and previous code reviews and suggested the reviewers for you? Not only familiarity with the code, but it could take into account the desire by engineers to familiarize with the code. One idea is that engineers mark the areas of the code that they'd like to familiarize and they'd be included automatically.

  • Rate reviewers: reviewers are different. Some provide high level analysis of the change, while others focus on the low-level details. What if the code review tool allowed you to rate the reviewers, and based on these reviews, it suggested the right set of reviewers to get your code well reviewed (from both high and low level perspectives).

  • Reasoning for changes: one of the first challenges to review is to understand the reasoning for the change and for the chosen solution. Oftentimes this info does not fit the comment section of the code and, as a reviewer, spend a code review interaction trying to clarify that. What if the code review tool required link to the design document and updated the design document with code review for that feature? Or if the design document is not available, it allowed details to be inserted into the review before it is sent out? Also, one could link bugs and add additional links that help providing context. Finally, tools could help prioritize what the main concern of that CR is - by allowing the requests to be specified and linked to the actual code related to it (e.g. "please focus on the correctness of this algorithm").

Update (August 07, 2012)
  • Automatic section coloring: based on the number of changes, on bugs, our intelligent code review tool could automatic color sections that it sees that can be more problematic. For instance, if a bug indicates where the problem is, that would raise the score for that file/section. If a bug fix touches that, the same thing. It can also take into account comments for that region in previous code reviews. With some heuristic combining this information, this tool could point risky sections automatically and also take some feedback from reviewers, which can rate the sections in terms of risk to help future code reviewers.