Hardcore Engineer, Wanderlust, Amateur Cook, Codes for Living IBM India Software Labs. Fitness Freak, Startup & Growth Hacker, Technology & Photography Enthusiast. Loves to get Lost.

Monday, December 28, 2015

The Scalability Challenge

Last week I was lucky to take out some time for leisure this Christmas Season wherein, I was catching up on all the cool stuff uploaded in 2015. That's when I learned about the 360 degree video camera and quite honestly I was stunned by this new phenomenon. I saw a Swiss Airbus flying over mighty and glorious Alp shot from the cockpit in 360 degrees. I could not stop myself from wondering about the fact that we indeed live in most wonderful times of history of mankind wherein we could witness, feel and do things which was, not too long ago was deemed impossible.

But with new internet technologies and products, comes new problems. And one of the most compelling problem for internet is scalability. I'm absolutely in love with the modern scalability problems, and very excited to address them head on. Although the world is full of "scalability engineers" who would die from an orgasm if their software ever saw 10,000 requests in a day, Scalability still remains my favourite forte.   

Talking about some numbers, Today internet and its components are hitting new records with every passing second. For instance at the time of this writing as of Dec 2015, Google in hitting close 1.2 trillion searches per year, They have a database of 120 trillion known URLs, 100 billion facts in knowledge graph, 500 hours of video uploaded every minute,  5 billion containers, 15 trillion Cloud Datastore ops/month. Not only Google, LinkedIn has 5.3 million hits second, 2 billion Tinder ratings per day, Twitter gets Billions of events every day, Android has a billion active users. 

One obvious question which one wonders looking at these numbers is, How are these companies, start-up doing it, and managing to make profit quarter by quarter? Before I dwell into details, I'd like to tell you two basic things one, that unlike milk, software gets more expensive, per unit, in larger quantities. second, the whole internet is dealing with the sliding window problem or as some like to call, the moving treadmill. To stay on the same spot, you have to run faster, every minute. For instance. the Google backbone carries the equivalent of 10 percent of all the traffic on the global Internet. The rate at which that volume is growing is faster than for the Internet as a whole. Which means, to constantly evolve is the only way to stay alive. The good old technique is not only true for living beings but even for technology architectures.
Michael T. Fisher, The highly acclaimed author of the famous book The Art of Scalability, once said, 'Describing architectures through implementation is akin to constructing a picture of your current or desired soulmate from pictures cut out of US Magazine; the result may paint a good picture of what you have or want, but it in no way describes how it is that the soulmate will meet your current or future needs'.  Which in a way means, you can plan all you want but it hardly tells you anything about the peak hour systems behaviour. Today as of 2015 downtime on internet is *expensive*, both in terms of loss of income and brand perception. Customers are quickly frustrated by any dips in service and are very vocal on social media in expressing that disappointment! Clearly, in an environment such as this, availability is a primary requirement. 

In addition companies want to ensure that they can meet the growing demand, They are constantly under pressure to introduce new services that move the customer experience forward.  As you'd expect, introducing new services places even more demands on an already highly complex system and there is only so hard you can push. 

First, as an application developer you need to learn to think "atomic". Atomicity is the key to highly concurrent applications. High concurrency is the key to high scalability. 

Fundamentally, one a basic level if you are building a service which supports around million transactions per day.

Breaking it in tiers would make be easier to understand

Tier 1 : Web Tier
Tier 2 : API Layer
Tier 3 : Data Source

And each of these tiers have thousands things you can do to enable scale, Technology menu catalogue is full of awesome and elegant technologies, ranging from programming languages, high throughput application servers, caching system, in-memory stuff, NOSql and micro-services. To scale up versus to scale out. It all depends on the specifics. You'll follow a distributed & scalable architecture and design from the very beginning. It is a would like to go in details of each and every one of them in my subsequent posts. 

But as general rule of thumb, the more dynamic or user-specialized your service is, the more difficult it is to scale. Facebook or Amazon are harder to scale than YouTube, which is harder to scale than Google, which is harder to scale than Wikipedia. 

As an Software Engineer in a big corporation, gives me good share of exposure to variety of scalability problems. I look at it this way, Architects look at thousands of buildings during their training, and study critiques of those buildings written by masters. In contrast, most software developers only ever get to know a handful of large programs well - usually programs they wrote themselves, and never study the great programs of history. As a result, they repeat one another’s mistakes rather than building on one another’s successes. So rather than believing that what you have is the most optimal, it is always good to have it run through the experts. I get my share of exposure to problem through Toptal Software development community I'm constantly up for new and challenging problems which you and your company might have. If you have a arhitectire, performance and scalability related requirement, I'd be more than happy to work on it. Please feel free to reach out at vikvikvik007 {at} gmail {dot} com 

Tuesday, June 26, 2012

Library.nu - The greatest site ever?

According to many, Library.nu was one of the greatest site ever build on the modern day web, if the reports were to be believed, Library.nu offered between 400,000 and a million digital books for free. Library.nu, previously called ebooksclub.org and gigapedia.com was a popular linking website, the knowledge sharing site along with elaborate review, instructions and snapshots. It was the largest repository of ebooks on the internet and better than any torrent site and a boon for students and research scholars not born with a silver spoon in the mouth. 

At the request of 17 publishing companies in the United States, the United Kingdom and Germany, including HarperCollins, Oxford University Press and Macmillan, a Munich judge granted injunctions against illegal posting or sharing of online book files by two websites in February 2012.

The thing that is unique about this is that library.nu offered access to books that were nearly inaccessible through any other means (such as out of print) or were copies of scholarly works that cost hundreds of dollars to purchase through legal avenues.

In a way this is like losing the Library of Alexandria all over again because many of these works will once again be seen by only a handful of people who either have fortunate access to them or the money to buy them.


Library.nu was of high importance for students and researchers who need rapid access to books specially in countries which the publishers do not sell and offer their books directly.

And not just any books - not romance novels or the latest best-sellers - but scholarly books: textbooks, secondary treatises, obscure monographs, biographical analyses, technical manuals, collections of cutting-edge research in engineering, mathematics, biology, social science and humanities.

The texts ranged from so-called "orphan works" (out-of-print, but still copyrighted) to recent issues; from poorly scanned to expertly ripped; from English to German to French to Spanish to Russian, with the occasional Japanese or Chinese text. It was a remarkable effort of collective connoisseurship. 

The book download portal Library.nu and cyberlocker ifile.it appear to have ‘shut down’ voluntarily after a coalition of book publishers managed to get an injunction against the two sites. According to the complaint, the sites offered users access to 400,000 e-books and made more than $11 million in revenue in the process.

According to Torrentfreak, "the legal team of the publishers estimated the revenue based on page impressions as well as estimated income from premium accounts, but this figure is laughable according to the iFile.it owner, which makes sense considering the site’s modest size." The iFile.it owner also told Torrentfreak that "they can barely cover the server costs with the revenue they make."

As a result, both sites have voluntarily pulled their services offline. Library.nu initially redirected to Google books and ifile.it has put up a message stating “no upload servers currently available.” and now Library.nu hosts Amazon books saying the old database has been deleted and iFile.it still working as usual for registered users.

The website (but not the database of ebooks) came back online 2 months later with an short message at http://library.nu/lnu.html claiming "most of the other articles in the media contain unhealthy amounts of speculation, and are not based on facts and reality.

But library.nu shared books; it did not sell them. If it made any money, it was not from the texts themselves, but from advertising revenue. As with Napster in 1999, library.nu was facilitating discovery: the ability to search deeper and deeper into the musical or scholarly tastes fellow humans and to discover their connections that no recommendation algorithm will ever be able to make. In their effort to control this market, publishers alongside the movie and music industry have been effectively criminalising sharing, learning and creating - not stealing.

Users of library.nu did not have to upload texts to the site in order to use it, but they were rewarded if they did. There were formal rules (and informal ones, to be sure), concerning how one might "level up" in the library.nu community. The site developed as websites do, adding features here and there, and obviously expanding its infrastructure as necessary. The administrators of the site maintained absolute control over who could participate and who could not - no doubt in order to protect the site from skulking FBI agents and enthusiastic newbies alike.

According to a russian academician Sonia useinova who says "I teach at a leading university in Russia, I barely survive on my salary and with an average price of a monograph in my field of studies being well over $50 there's no question of buying anything. No foreign scientific literature finds its way in our library, so library.nu was one of the few ways I could find of reading books I have to. now it's gone and it's really a blow.
As for royalties, I happen to know from a colleague, world-famous scientist, that Brill (the most important publishing house specializing in oriental studies) had paid him a staggering $300 for publishing a monograph they sell at $98 a copy. The idea is you should be proud they consent to publish your stuff. 
So it's down to wanting to learn but not being able to afford the books. "

According to many who believe that such incidents are why companies like Elsevier & Springer are evil profiting hugely upon the work of academics. Read more at : http://en.wikipedia.org/wiki/The_Cost_of_Knowledge

Many talented developers around the world who once benifitted from the site are planning to write a Library.nu aleternative/clone and according to some the development discussions have already started over IRC on channels like #librarynualt.

Library.nu Alternatives

Do share any other good Library.nu alternatives which you might know.  

Monday, June 18, 2012

Amazon Web Services in a Nutshell

  • Amazon Elastic Compute Cloud (EC2) :   Amazon Elastic Compute Cloud delivers scalable, pay-as-you-go compute capacity in the cloud.   Pay only for what you use. There is no minimum fee.  Renting a server per hour, root access. Define the size your self and pay for it. Learn more HERE
  • Amazon Elastic :  Amazon Elastic MapReduce (Amazon EMR) is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).   Learn more  HERE
    • MapReduce    Renting a multiple servers per hour e.g Hadoop cluster (to crunch a lot of data)
    • Auto Scaling   Renting just enough servers as required (essentially, bidding system, using more with a lot of work, using less with less work)
  • Amazon CloudFront :   Economical Personal CDN, content distribution network.  Amazon CloudFront is a web service that makes it easy to distribute content with low latency via a global network of edge locations.  Learn more  HERE
  • Amazon SimpleDB : It's a noSQL for key-value store. Quickly searchable. Amazon SimpleDB is a highly available and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests and Amazon SimpleDB does the rest. Learn more HERE
  • Amazon Relational Database Service (RDS): MySQL as a Service, one of the best services of AWS.  Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud.  Learn more  HERE
  • AWS Elastic Beanstalk :   Automated Infrastructure for java/PHP/… applications. Works with tomcat, uses internal services as required (turn-key solution, Good to start with for AWS services.)  AWS Elastic Beanstalk is an even easier way for you to quickly deploy and manage applications in the AWS cloud. You simply upload your application, and Elastic Beanstalk automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling, and application health monitoring.  Learn more  HERE
  • AWS CloudFormation :    Automated setup for service/app/platform you want to use. Choose a service, give basic details and CloudFormation will do the rest. best for WP blog, CMS etc.  AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.  Learn more  HERE
  • Amazon Fulfillment Web Service (FWS) :    Amazon manage your fulfillment. your suppliers ship to Amazon, and they ship to your customers. (AFAIK, US ONLY)  . Learn more  HERE
  • Amazon Simple Queue Service (SQS) :   ‘first in, first out’ messaging queue. perfect for jobs that should be processed asynchronously, more or less in order of arrival. Lazy tasks video rendering, thumbnails .  Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers.  Learn more  HERE
  • Amazon Simple Notification Service (SNS) :   PUBSUB (publish/subscribe mechanism). You create topics interested parties can subscribe to. sending a message to a topic, sends the message to all subscribers. (Like a mailing list, but for software systems.)   Amazon Simple Notification Service (Amazon SNS) is a web service that makes it easy to set up, operate, and send notifications from the cloud.  Learn more  HERE
  • Amazon Simple Email Service (SES) :    Service for sending mail. But you have to earn your quota with good behavior.  Amazon Simple Email Service (Amazon SES) is a highly scalable and cost-effective bulk and transactional email-sending service for businesses and developers. Amazon SES eliminates the complexity and expense of building an in-house email solution or licensing, installing, and operating a third-party email service. The service integrates with other AWS services, making it easy to send emails from applications being hosted on services such as Amazon EC2.   Learn more  HERE
  • Amazon CloudWatch :    Monitoring of other AWS assets.  Amazon CloudWatch provides monitoring for AWS cloud resources and applications.  Learn more  HERE
  • Amazon Route 53 :  DNS (Awesome), Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. Learn more  HERE 
  • Amazon Virtual Private Cloud (VPC) : Extend your private network with ‘some cloud’.  Amazon Virtual Private Cloud (Amazon VPC) lets you provision a private, isolated section of the Amazon Web Services (AWS) Cloud where you can launch AWS resources in a virtual network that you define. With Amazon VPC, you can define a virtual network topology that closely resembles a traditional network that you might operate in your own datacenter.  Learn more  HERE 
  • Elastic Load Balancing : Distributing traffic over one or more instances (servers).
    Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. Learn more HERE

  • Amazon Simple Storage Service (S3): Almost unlimited storage, best for Videos, Images etc. Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.
    Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. Learn more HERE
  • Amazon Elastic Block Store (EBS) : Network disks for AWS. (Attach a volume to an instance of your choice. you can’t share the disk between multiple instances).  Amazon Elastic Block Store (EBS) provides block level storage volumes for use with Amazon EC2 instances.  Learn more HERE.
  • AWS Import/Export:  Export  and Import large amounts of move large data in and out of S3.  AWS Import/Export accelerates moving large amounts of data into and out of AWS using portable storage devices for transport.  Learn more HERE.
  • Amazon Mechanical Turk : Human powered Threads, work by the task. you want to have 10,000 users visit your site and try to do something. put those tasks on mechanical turk (for $0.1 per task?) and any one can help you. Learn more HERE
  • Identity and Access Management (IAM): Let’s you provision users within your AWS account and grant them access to different services and resources. Learn more HERE.

Tuesday, December 27, 2011

Best ePub Readers For You

ePub(also referred as EPUB or EPub) stands for Electronic Publishing and is a standard e-book format which is used by all popular e-book readers out there, including Barnes & Noble Nook, Sony Reader, iPhone/iPod Touch(using various e-book apps), and more. ePub files have .epub extension.

There are various online as well as desktop tools which allow users to read .epub formats. We have covered four ePub readers which we believe you all should know about.

  • Bookworm is an online service that aims to be become a leading free platform for reading ePub books online. It is part of O’Reilly Media. It takes only three steps, create an account, upload an ePub book or magazine, and start reading.
  • The benefit of this service is that you can read the e-book online no matter where you are located and which device you are using.

  • EPUBReader is a handy add-on for Firefox that can directly open any ePub format quickly. It is indeed faster and quicker than desktop software and shows the graphics far better than bookworm service.
  • This add-on opens the ePub file just like you can open PDF file in Firefox browser. Just hit the download link of any ePub book and select Open. If you have already downloaded the ePub book, then you can open it with Firefox.
  • The interface is customizable, which means you can change the background colors, etc to your own liking. The navigations buttons are given at the bottom along with the option to change the fonts.

  • Stanza is a desktop ePub reader and converter that is fast, light-weight, and simple. It can be used to save bookmarks and quick conversions. You can convert .epub format into tens of different other document formats including doc and pdf formats. If you want to read the ePub file in Amazon Kindle, there is an option for that as well.
  • The only downside is formatting and navigation, the two most important aspects. For formatting it can be compared with Bookworm service but is not as great as EpubReader add-on for Firefox.

  • MobiPocket Reader is not only an advanced ePub books reader but is also a great professional books organizer. You can import books in PDF, ePub, or Office formats and organize them quickly. Just double-click a book and start reading.
  • The formatting of MobiPocket is better than the above mentioned tools and has an option to zoom into the book, useful for those who want to read the tiny text on the book. Also there is an advanced search box given on the top-right side which can drastically help in saving time.

Update: Also check out FBReader, which is an outstanding eBook reader, manager, and downloader.

Update 2: As a side note, you will also be interested in PaperCrop. A free tool that allows optimization of PDF file so that it takes less space and opens faster on mobile devices.

For quicker distraction-free reading, I found EPUBReader add-on to be the best. For traveling, I would prefer using Bookworm since all uploaded books can be accessed from anywhere in the world. For converting ePub books and reading them without worrying about Firefox crash, I would prefer Stanza. Lastly, if I am looking to organize all the books in one place, then MobiPocket Reader is the app to of choice.

Tuesday, December 20, 2011

We Love Developers

         An Awesome Watch for all you Technology Folks!

Monday, November 21, 2011

Hidden Tips & Tricks for VLC Player

VLC is the most popular media player today, and overtime it has emerged as play it all player, you just name it, it will play it for you. But VLC can do a lot many things as well as which normal user do not know about. I will list couple of common hacks for VLC, you can append the list if you think if anything can be added: 

  • Rip DVDs: VLC includes a basic DVD ripper. You probably would never use it when there are better DVD rippers available, but it helps to know that you can in fact, get a decent quality DVD rip with VLC. To rip a movie follow these steps: Go to the Media menu and choose Convert/Save. Click on the Disc tab.

  • * Here you can adjust the Starting Position and rip only specific titles or chapters.
    * Enter file name making sure to end with .MPG, and start ripping.
    * Click Save.
    * Its Done.
  •  Record videos: With the new VLC, you can record videos during playback. The record button is hidden by default. To see it, click on View>Advanced Control. The record button will now appear. Clicking on the button while playing a movie or video will start recording. Clicking again will stop recording.
  •  Play RAR files: Do you know VLC can play videos zipped inside RAR files? They play like normal video files and you can even use the seek bar. If the RAR file is split into several files, no problem. Just load the first part (.part001.rar ) and it will automatically take the rest of the parts and play the whole file.
  •  Play in ASCII mode: VLC media player has an amusing ability, to playback movies in ASCII art. To enable ASCII mode, open VLC media player and click on Tools>Preferences. Open the section “Video” section and under “Output” select “Color ASCII art video output” from the drop down menu. Save it. Now play any video file to enjoy the ASCII art.
  •  Listen to online radio: VLC includes hundreds of Shoutcast radio stations. You just need to enable it through Media>Services Discovery>Shoutcast radio listings. Now, open the Playlist and browse through the stations.
  •  Convert Audio and Video formats: In VLC you can convert video and audio files from one format to another. Several different formats are supported like MP4, WMV, AVI, OGG, MP3 etc. To access the converter:

  • * Go to Media>Convert/Save.
    * Load the file you want to convert using the Add button and click Convert.
    * Now choose the output format and output file location.
  • Download YouTube and other online videos: First grab the URL of the YouTube video page. Now click on Media>Open Network stream. Paste the URL and click Play.
  • Once VLC starts streaming the video, click Tools>Codec Information and at the bottom of the window you will see a Location box. Copy the URL and paste it on your browser’s address bar. The browser will now download the file which you can save it to your hard disk. Alternatively, you can record the video.
  • Streaming the screen shot of your desktop:
    1.Open VLC media player
    2.Press ctrl+N
    3.type 'screen://' (without quotes)
    4.click on PLAY
    Now wait n watch ;)

Wednesday, October 12, 2011

The Era of Steve Jobs!

On the uneventful morning of Thursday, 7.20 AM IST, I was in deep sleep at my apartment in the downtown Bangalore as it was a holiday on the occasion of some festival in India, I was brought to life by my ringtone ringing aloud, I reached for my Blackberry, It was Sanjeev calling, It was very unlikely for him or for that matter anyone to call me this early, if not for anything fatally important, I sensed there was some problem. As soon as I picked he just spoke one sentence, "Vikas you know what? Steve Jobs is Dead!" 
 I will remember that very minute – still lying on my bed, I was staring the wall clock ticking in front of me, Suddenly, everything went out of focus. I could hear the blood pounding my head; tears welled up in my eyes.

It is perhaps the only time that I didn't care for checking out the news and what people have to say about him on my Twitter feed; I didn't want to discuss it with anyone or Tweet about it, why doesn't the world realize that my Elvis is dead! I don’t care about news. I don’t care about a world that is a lot less exciting than it was when Steve was around. Can’t you see that my soul is being put through a meat grinder.

Every generation has its heroes. I was too provincial to love the Beatles and cry over John Lennon. I was too Indian to care much about Elvis. And I read about President Kennedy in books. But for me, Steve Jobs was all of those people. I don’t know why, how and where that happened but Jobs was my icon.

For many of us who live and die for technology and the change it represents, he was an example of what was possible, no matter how the chips were stacked against you. Jobs put life and soul into inanimate objects. Everyone saw steel, silicon and software; he saw an opportunity to paint his Mona Lisa. People saw a phone; Steve saw a transporter of love. People saw a tablet; he saw smiles and wide-eyed amazement. They made computers; he made time machines that brought us all together through a camera, screen and a connection.

Mac, iPod and iPhone — they are like every technologist ’s dream gadget- they are magical, memorable and life-changing. We really have to understand that weather we use an Apple product or not, whatever gadget we like, to some extent there is a large enough impact from the innovation which Steve brought in the form of his products. Even the ones who used to criticize Steve at some point agreed that its an irreparable loss to the entire humanity.

The idea of Steve led me to follow my heart, make tough choices, be brutally honest with myself (and sometimes annoying to people I love) and always remember that in the end, it is all about making your customers happy. There are simple ways to get along with everyone. There are easier ways to get things done. There are compromises. But to me Steve Jobs meant try harder, damn it, your customers (readers) expect better than that. Steve taught me to care about the little things, because in the end, little things matter.

Steve was my secret muse. Trust me –- he is a secret muse to many of us around the world. Mark Zuckerberg. Jeff Bezos. Dave Morin. Jack Dorsey. We are all part of the tribe called Jobs. There is a whole generation of entrepreneurs who ask themselves this one question –- what will Steve do. Natch. What would have Steve done!