Friday, November 23, 2012

Career Advice for Software Engineers

A great career advice post for software engineers. It definitely matches my experiences and my observations:

Don't Call Yourself A Programmer, And Other Career Advice

Find bellow an excellent quote from this post. Sad, but true reality:
The person who has decided to bring on one more engineer is not doing it because they love having a geek around the room, they are doing it because adding the geek allows them to complete a project (or projects) which will add revenue or decrease costs. Producing beautiful software is not a goal. Solving complex technical problems is not a goal. Writing bug-free code is not a goal. Using sexy programming languages is not a goal. Add revenue. Reduce costs. Those are your only goals.

Thursday, November 22, 2012

Success: not features, but solving customer's problems

Not all places are very customer focused due to culture and/or inefficiencies, but I have to say that this quote resonates with me:
"Success is not delivering a feature; success is learning how to solve the customer's problems" - The Lean Startup
Working for the customer can be quite motivating for employees.

Tuesday, November 20, 2012

Using queues for asynchronous processing

Great article on the reasons to consider using message queues for asynchronous processing. I do remember the hard time I had to convince coworkers of the advantages of queue - people that never used them are reluctant to use them and very often prefer to shoehorn solutions on top of databases. That is what this article talks about:

Asynchronous Processing in Web Applications, Part 1: A Database Is Not a Queue

Saturday, November 10, 2012

Generating Sequential IDs

A typical problem we see in distributed systems is how to generate sequential IDs efficiently. Here you can find three approaches:

Percolator (Google)

The timestamp oracle is a server that hands out timestamps in strictly increasing order. Since every transaction requires contacting the timestamp oracle twice, this service must scale well. The oracle periodically allocates a range of timestamps by writing the highest allocated timestamp to stable storage; given an allocated range of timestamps, the oracle can satisfy future requests strictly from memory. If the oracle restarts, the timestamps will jump forward to the maximum allocated timestamp (but will never go backwards). To save RPC overhead (at the cost of increasing transaction latency) each Percolator worker batches timestamp requests across transactions by maintaining only one pending RPC to the oracle. As the oracle becomes more loaded, the batching naturally increases to compensate. Batching increases the scalability of the oracle but does not affect the timestamp guarantees. Our oracle serves around 2 million timestamps per second from a single machine.


Snowflake (Twitter)

To generate the roughly-sorted 64 bit ids in an uncoordinated manner, we settled on a composition of: timestamp, worker number and sequence number. Sequence numbers are per-thread and worker numbers are chosen at startup via zookeeper (though that’s overridable via a config file). We encourage you to peruse and play with the code: you’ll find it on github. Please remember, however, that it is currently alpha-quality software that we aren’t yet running in production and is very likely to change.


MySQL Ticket Server (Flickr)

A Flickr ticket server is a dedicated database server, with a single database on it, and in that database there are tables like Tickets32 for 32-bit IDs, and Tickets64 for 64-bit IDs.

The Tickets64 schema looks like:
CREATE TABLE `Tickets64` (
  `id` bigint(20) unsigned NOT NULL auto_increment,
  `stub` char(1) NOT NULL default '',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `stub` (`stub`)
SELECT * from Tickets64 returns a single row that looks something like:
| id                | stub |
| 72157623227190423 |    a |
When I need a new globally unique 64-bit ID I issue the following SQL:
REPLACE INTO Tickets64 (stub) VALUES ('a');

Romney's Orca fiasco

This is a great article on the IT project fiasco in Mitt Romney's campaign:

Inside Team Romney's whale of an IT meltdown

This is a good quote that is representative of much you see around among IT projects:
[...] I had some serious questions—things like 'Has this been stress tested?', 'Is there redundancy in place?', and 'What steps have been taken to combat a coordinated DDOS attack or the like?', among others. These types of questions were brushed aside (truth be told, they never took one of my questions). They assured us that the system had been relentlessly tested and would be a tremendous success."

Sunday, November 04, 2012

Consensus Protocol

You probably heard of 2-phase commit, Paxos, consensus protocol. Paxos itself seems to be a complex thing to understand and, although proven to be correct, it's not uncommon to hear that it's not really required and some "simpler" algorithm can be used.

These posts I came across give a great explanation on the shortcomings of 2-phase commit when it comes to failures and also of 3-phase commit, which handles well one type of failure (fail-stop). It gives great examples of hard distributed system issues with these protocols that make them not robust enough.

And this is a great quote from these articles:
Mike Burrows, inventor of the Chubby service at Google, says that “there is only one consensus protocol, and that’s Paxos” – all other approaches are just broken versions of Paxos.
I'd definitely recommend you take the time to read them:

Consensus Protocols: Two-Phase Commit
Consensus Protocols: Three-phase Commit
Consensus Protocols: Paxos

And, just to finalize, when I see quotes like above and the number of issues with 2PC and 3PC, I wonder how reliable consensus protocols like the one used by MongoDB to pick the primary replica actually is:
We use a consensus protocol to pick a primary. Exact details will be spared here but that basic process is:
1.get maxLocalOpOrdinal from each server.
2.if a majority of servers are not up (from this server's POV), remain in Secondary mode and stop.
3.if the last op time seems very old, stop and await human intervention.
4.else, using a consensus protocol, pick the server with the highest maxLocalOpOrdinal as the Primary.

Any server in the replica set, when it fails to reach master, attempts a new election process.

Data replication: why 3 copies?

If you ever asked yourself why these storage systems talk about making 3 copies of your data, check out this article that came out in July and that I just reread:

Data Replication in NoSQL Databases

It talks about RAID, why it's not used for big databases, and the reasons for this magical 3 for data replication.

Note that the same reasoning could apply to any replication you may be doing yourself, even at the application level.

Saturday, November 03, 2012

NAT, peer-to-peer and hole punching

This is a great article if you want to understand how connections directly to your game console behind your router (NAT) happens:

Peer-to-Peer Communication Across Network Address Translators

As it turns out, although SYN packets for TCP connections are blocked by default, there are techniques that "punch holes" in the NAT and allow them to go through. For services like Xbox Live or Skype, for instance, that minimizes the need of relay servers.