ID2020 Summit

Yesterday, I attended the ID2020 summit held at the UN and at Microsoft office near Times Square. It was a great meeting of folks from humanitarian and private sector focused on identity as a fundamental human right.

I thought the following points stood out (obviously with my bias :) ):

Identity is a fundamental human right.

The ID2020 conference was about that obviously, but come to think of it, is there anything that can be done in the world today without an identity? Getting a mobile phones, which is probably the universal device every one has, requires an identity.

Human identity needs to be decoupled from national identity

If identity is a human right like food or water, then it should probably be decoupled from national identity. Could there be a truly globally distributed identity system? But is that even possible? And is that really needed? A government issued identity probably is more valuable to a farmer than a fully distributed identity. Obviously, food for thought.

Vaccination card is often the first identity of a human being

Almost every child in the world is vaccinated. A vaccination card is often the first identity for a child in a world where birth certificates do not exist.

Simplicity is the king - for identity too

If there is one lesson from Aadhaar, India's identity system for a billion people, it is that simplicity is the king. An identity system that tries to meet requirements for everyone cannot work. Instead, a simple system which provides citizens with a digital identity that is verifiable, and which allows application to be built on top using derived identities is probably the way to go.

Of course, in a system like Aadhaar, the issuing authority is the government. What if the government turns on its citizens?

A human should own their identity

There is a reason phrases in English exist such as, "I have my own identity". A human should own their identity and the data associated with it, and provide consent to it.

Identity is for things too

While identity for humans is important, the things in the "Internet of things" need an identity too.

Verification / attestation of digital identity

When identity goes digital, the need for verifying identity arises. Aadhaar, for example, has 600 Million verifications per month. Similarly, identity attestation is needed for various applications


The above list is just my recollection; there were many interesting discussions in different focus groups.

Onwards to digital identity.

Solving Developer's Security Challenges Through A Secure DevOps Pipeline (SDO)

The previous post described the challenges a developer faces in securely deploying the applications and managing their steady-state security. The difficulties arise, every security and compliance capability needed by developers is not available as a service consumable through an API (e.g., invoking an API to start a penetration test). As a consequence, the developer has to determine all the details of security and compliance for its application, which is no easy task.

Before we understand various security and compliance capabilities needed by developer, we must understand what is the life cycle of an application.

Life Cycle of an Application

An application life cycle comprises, development and build, deployment and testing, running and updates, and finally decommissioning. The following figure shows a typical application life cycle.

We also refer to this life cycle as the DevOpos pipeline. In an ideal DevOps pipeline, developers develop their applications, package their application components and dependencies using technologies such as Docker to create "immutable code", deploy them using templates such as Docker Compose, run them in a cloud, deploy updates continuously, and may ultimately decommission an application, partly or entirely.

DevOps Tension with Security and Compliance

This "continuous development and deployment" cycle creates tensions with the conventional security approaches. For example, applications comprising multiple components (e.g., webserver and database server) need "keys" or "passwords" for communication, yet these "keys" and "passwords" often end up in continuous integration tools such as Jenkins or Travis often with limited or no access control, or worse in code. Moreover, the "immutable code" images may end up in public repositories. Worse, it may be a requirement to not have any security credentials as part of the "immutable image"; such credentials must only be available at run-time. Also, source code scan, network and malware scans may not be performed before every continuous update, leading to security holes.

Secure DevOps Pipeline

How can a system possibly alleviate the difficulties developers face in securely deploying applications and managing their steady-state security? The answer is to convert every possible security function, whether it is a capability or an advisor into a service that is easily consumed by a developer with little or no effort from the developer. If achieved, this is the ultimate "Secure DevOps Pipeline", where every security and compliance function is consumable through an API.

So what are the security and compliance capabilities needed in a Secure DevOps Pipeline? Here is a pictorial rendition of key capabilities needed in such a pipeline:

tl;dr

Secure DevOps - Develop and Build Phase

Application development typically starts with a problem statement, design, and initial prototypes. The prototypes and subsequent revisions targeted for production comprise code written by application developers. The code may also have dependencies such as libraries, dependent packages, and so on.

TRUSTED CODE capability provides an implies that all application code and its dependencies are from trusted repositories.

The source code written by application developer is stored in trusted and managed repositories. Such repositories may have been configured with some automatic code scans on every code commit.

When an application developer builds an immutable image (Docker image) for its application source code and dependencies, and may run package installation tools such as apt-get install XYZ, the dependent packages may also get installed. If the version and source for those dependent packages is not controlled, old packages with potential vulnerabilities get installed or worse malware  may creep in. The job of the "Trusted source" capability is to ensure that application dependencies are from known or trusted sources.

In image formats such as Docker images, a developer has complete control over which repositories to use. Without any restrictions, a developer may download code or dependencies from anywhere, leading to the first chink in the security posture: is the code even trusted? The job of the "Trusted Code" function is to apply restrictions on the source and version of application code as well as dependent packages.

Trusted code is easier said than done. As mentioned above, it involves restricting developers to a set of trusted repositories, not all of which may satisfy dependencies for an application. However, verifying code / package and dependencies and adding them to trusted repositories is a continuous process. Overtime, the curated repositories will contain packages that should satisfy developers needs.

CODE SCAN capability scans the source code written by developers to ensure that it complies with security best practices. This capability is especially needed if "Trusted Code" capability is not available.

Developers are lazy. They will typically take the shortest route to complete their task. Without adequately enforcement of code best practices, errors may creep in the code that lead to security holes, e.g., SQL injection.

CODE / PACKAGE VULNERABILITY determines vulnerabilities in dependent code used by an application. While source code written by developers may get appropriate scanning as part of "Code Scan" service, the vulnerabilities in dependent code / packages need to be identified through such a service.

An application may have numerous dependencies that are satisfied through numerous package management tools. Typically, such package management has been associated with utilities such as "apt-get install" or "yum install". However, modern applications make use of several package managers such as python pip or Node.js npm. Worse, package managers may not always exist, e.g., .jar packages available on bintray. While operating system distributions such as Red Hat or Ubuntu regularly publish security bulletins for packages published in their repositories, the same cannot be said for every package manager. Consequently, it may not be possible to accurately determine vulnerabilities associated with each package.

Nevertheless, such a service is crucial in identifying known vulnerabilities.

APPLICATION CONFIGURATION VALIDATION is required to determine the correctness of application configuration.

Modern applications are complex. They comprise hundreds and sometimes thousands of configurations. Getting application configuration right from security perspective when it is deployed through layers of automation is extremely hard.

An application configuration validation is a service that validates the application or its component configuration from a security (or potentially performance) perspective and alerts the developer of incorrect configurations before an application is deployed into production.

MALWARE / VIRUS SCAN is a service that determines if any malware or virus has crept into the immutable image of an application build.

LICENSE VERIFICATION is a service that validates the license of application components being deployed.

License verification is needed for two reasons. First, the service ensures that only components with known licenses are deployed. For a capability delivered as a service, this is less of a concern since the software components having most restrictive software licenses such as GPL can potentially be used in delivering the service.

The other reason for license verification is appropriate charge back. For traditional enterprise software, it is important to determine the appropriate licenses for software being deployed. However, charge back for deployed software becomes less of a concern as traditional software is increasingly delivered as a service.

DEVELOPMENT / AUTOMATION CREDENTIAL MANAGEMENT is critical in ensuring good security practice. Building an immutable image for an application is often done through automation (Jenkins and Travis). Ensuring appropriate accesses for such credentials and key management is critical for good security hygiene.

Similar to user authentication, authorization, and access, any credentials required for automation must be kept in a credential or key store which is delivered as service.

Secure DevOps - Deploy and Test Phase

Application deployment and testing happens continuously and iteratively with development, and eventually running. Following key capabilities during deployment and testing, delivered through an API, can help reduce the "security" burden on a developer.

APPLICATION PATTERN allow developers to specify how various components of an application are combined together, typically over network, in delivering the application function. Such specification is then run by an deployment engine.

Cloud platforms provide numerous ways of specifying a template for application deployment. These templates vary in order of flexibility and ease of use. Typically, the templates that are highly flexible make it onerous for developers to specify correct security properties.

From a security perspective, the application deployment specifications need to follow "secure by default" principle while providing flexibility to the developers to override any details.

APPLICATION CREDENTIALS AND KEYS need to be managed similar to user credentials or deployment tools credentials.

Applications comprise multiple components such as web server and a database server (see this example) or even remote services delivered through APIs. Credentials are required to communicate among components or remote services. Such credentials, referred to as application credentials and keys, must also be stored in key-management systems.

Thus, a key management system will store credentials for deployment automation as well as inter-component communication. Such credentials will likely vary across development, stage, and production pipelines.

These keys may also need to be periodically rotated, which places additional burden on the developer. Having a service which will automatically [re]-generate the keys and configure application or its components with newly generated keys can significantly help reduce the key management burden on the developer.

SECURE AUTO SCALING. Applications deployed in cloud need to scale as load increases. Ideally, this scaling is done in an automatic manner. Such scaling implies that portions of incoming traffic will be routed to a newly spun application component responsible for handling the traffic. Such new routing must also be secure. That is, any credentials needed must be added to the newly spun or decommissioned component at run-time.

MONITORING must be configured for an application components whether it is a fresh deployment, an upgrade, or auto-scale. Monitoring encompasses traditional metrics such as CPU, memory, disk, and network; application-specific metrics, and logs. Monitoring can be passive or active.

As part of monitoring configuration, malware and anti-virus may also be configured.

APPLICATION NETWORK SCANS and AUTOMATED PENETRATION TESTING. When an application is deployed or updated, appropriate network scans and penetration tests must be performed on it before exposing it to the general users. Typically, network scans make use of tools such as Nessus, while penetration tests are typically done through a manual process. These scans and tests can be automatically invoked upon a new deploy or commit in dev/staging/production.

With some aid from developer, the penetration tests can also be automatically performed, contributing to an automated secure devops pipeline.

ENCRYPTED / INTEGRITY STORAGE. Applications deployed in cloud may require the underlying storage to be encrypted or provide guarantees against tampering. A cloud may provide encrypted and integrity storage as part of its offerings. Application developer may configure the use of encrypted and integrity storage in an application pattern or otherwise.

DevOps personnel may specify the use of encrypted storage as part of application patterns. DevOps may bring their own keys or have the cloud auto-generate the keys for encrypting storage. Both types of keys need to be managed, similar to user and API keys.

NETWORK AND APPLICATION FIREWALLS / IDS AND APPLIANCES may also need to be deployed to meet regulatory compliance and good security practice.

In application pattern templates, a developer may indicate the use of compliance regime. The cloud can then automatically deploy and configure network and firewall appliances on behalf of the user.

Admittedly, deploying and configuring network and application firewalls is a "black art". It is often very difficult to get it right due to myriad configurations.

By following the "secure by default principle" and converting the most commonly used aspects of these appliances into functions delivered through APIs, the deployment and configuration of these devices can be integrated into a secure devops pipeline.

TESTING requires various components of an application, and the application as a whole to be continuously tested. The tests, unit, functional, or integration, must be written by developers and their invocation should automatically be done as part of build or deploy phases.

Secure DevOps - Run Phase

Run phase invokes certain capabilities of develop and build, and deploy and test phases in a continuous manner. These capabilities, explained earlier, include:

• CODE / PACKAGE VULNERABILITY SCAN
• MALWARE / VIRUS SCAN
• APPLICATION NETWORK SCANS
• AUTOMATED PENETRATION TESTS

Following additional capabilities are needed in the run phase.

APPLICATION / CLOUD CONFIGURATION VALIDATION. Once applications are deployed in cloud, the cloud configuration also needs to be validated. Cloud configurations encompass configurations of various cloud-based services such as firewalls, encrypted storage, key lengths, security groups, geographical distribution and so on. Thus, as part of running the application on cloud, both application and cloud configurations need to be validated together.

SCANNING FOR SENSITIVE INFORMATION IN LOGS. Sensitive information in logs such as personal health information (PHI) or social security numbers need to be scrubbed from application logs that may otherwise be viewed for debugging or administrative purposes. If best practices were followed for development, such information would not have ended in logs in the first place. Nevertheless, the scanning service may tag sensitive pieces of information in logs, which may need to be scrubbed or removed all together.

EVENT LOGGING AND MONITORING that was configured in the deploy and test phase must be monitored for any application events, malfunction, or incidents.

ENCRYPTION OF COMMUNICATION. The communication from end-users of an application of among various components of an application must be appropriately encrypted, meeting the security and compliance guidelines. This communication must be periodically monitored, especially along configuration changes, to ensure that communication that was encrypted using say high strength ciphers has not been downgrade to use a low-strength cipher as a result of an update.

If secure devops pipeline was completely followed, all application component communication will likely be encrypted or in isolated networks. The setup of encryption can be done as part of specification in application pattern.

INTEGRITY MONITORING. The integrity of data stored by an application must be monitored through APIs. Any update to the data must be recorded and be alertable so that appropriate actions can be taken.

OPERATIONS of service are fully automated, and changes are logged. These changes include bring up or tear down of services, logging for administrative purposes and so on.

Secure DevOps - Decommission

Eventually, part of an application component or an entire application may need to be decommissioned. Partial decommission may happen, for example, if the load on application decreases. Full decommission may happen if an application is no longer needed. As part of decommissioning, scrubbing of resources may be needed. These resources include:

LOG / INFORMATION SCRUBBING

PHYSICAL AND VIRTUAL RESOURCES 

Among virtual resource decommissioning, any keys that may have been used but are not longer needed must be appropriately deleted.

Secure DevOps - Policy and Verification That Puts it All Together

Since security features are delivered via an API as part of secure devops pipeline, there is a need for having a policy in place that checks for violation of security features, and a verifier engine that validates the results of various features along the secure devops pipeline.

In summary, delivering all security functions through an API, and making them readily consumable by developer is not easy. The ingredients that will make a secure devops pipeline possible are key management (user, automation, application), scanning (code, package, configuration, network, malware, virus), testing (penetration and functionality), patterns, logging (API, access), and authn/authz.

Can a Cloud Help Developers "Securely" Deploy, Run, and Manage their Applications?

Can a cloud help developers deploy applications securely in a cloud and manage their steady state security? In this three part series, I will discuss this question in detail.

This part covers the challenges a developer faces in deploying applications securely, and managing their steady-state security.

Challenges in Securely Deploying Applications and Managing Their Steady-State Security

Typically, cloud is associated with scale. If an entity desires to scale their application to serve hundreds of thousands of users, cloud is the answer.

Modern applications, deployed on cloud or else, are complex. The comprise multiple components, which interact with each other over network. Each component may or may not hold state, and typically delivers a unit of functionality (micro-services etc).

In cloud, a developer writes automation code to deployed and upgrade these applications, and may leverage cloud monitoring tools to manage the application. 

Security and compliance requires a developer to deploy applications according to the acceptable best security practice of that time. Conforming to these best practices requires substantial security knowledge and expertise on part of the developer.

As an example, consider a simple application comprising a web server and a database server as shown in figure below.

To ensure the security and compliance of this application, a developer has to ensure a number of steps:

1. Best Practices for Source code. The source code should conform to well-known security best practices. As an example, any SQL queries issued by the web server must not be prone to SQL-injection attacks.
2. Message Confidentiality. The communication between users of this simple application and the web server, as well as web server and database server must be encrypted.
   Implementing this functionality typically requires knowledge of appropriate TLS protocols and ciphers, creation and setting of PKI certificates and encryption keys, and configuring them appropriately. While services such as LetsEncrypt make it easy to generate certificates, it is still up to the developer to configure her applications correctly with them. Moreover, these certificates need to be periodically rotated.
3. Managing Application Keys, Certificates, and Passwords. Establishing 'trust' among various components of an application is done through keys, certificates, and passwords. While we typically associate passwords and certificates with end users, the components of an application must establish authentication and authorization among themselves to establish trust. The burden for creating, storing, and managing these credentials is on the developer.
4. Encryption of Data at Rest. Compliance regimes such as PCI and HIPAA may require the data at rest to be encrypted. A developer has to configure database servers to encrypt the data per record or per disk, and manage the keys associated with them.
5. Configuring Security Groups. A common feature of IaaS clouds is that they allow incoming traffic to an instance be limited to certain open ports on the instance. It is non-trivial to configure security groups correctly for an application comprising components running in tens of instances or more, due to complex interactions, high availability etc. 
6. Collection of Logs, and Scrubbing of Sensitive Information in Logs. The logs generated by multiple application components needs to be collected to meet security and compliance as well as operational needs. The developer needs to ensure that no sensitive information such as passwords of the users of its service are present in the logs.
7. Setting Up Intrusion Detection Systems (IDS) and Firewalls. A developer may need to setup intrusion detection and prevention systems to log any suspicious activity as well as any other firewalls. Typically, in IaaS clouds, security groups double as firewall, so a separate firewall deployment may not be needed.
8. Admin Access to Web and Database Servers. The developer or the operator of this simple application two tier needs to manage credentials for admin access to web and database servers.
9. Network Scan of Web and Database Servers. While a developer may have undertaken necessary steps to configure and deploy her application securely, it is still prudent to perform network scans before and after deployment, to ensure conformance to the intended model. These network scans can be as simple as port scans, or can also be application specific.
10. Penetration Testing of Application Components. Similar to network scans, penetration tests actively try to break the application as may be attempted in wild. Such tests are customized for the application (web server in the above example).
11. Secure Scaling of Application. When scaling web server or database server, the applications keys, certificates, and passwords need to be setup correctly and stored appropriately. Even in a simple application such as the one shown above, it is non-trivial to configure keys and certificates on load balancers or other components that are required for application to scale.

As it can be seen from this example, it is a non-trivial task developers to deploy and configure their applications securely and requires significant expertise on developer's part.

In the next post, I will describe the concept of a "Secure DevOps Pipeline", which can facilitate a developer in deploying and managing her applications in a secure manner.

In the company of giants - at IEEE Annual Award Ceremony

Who will not get inspired, sitting in the company of giants, and marveling at the accomplishments of all highly accomplished individuals who received their awards at the IEEE Annual Award ceremony, held at Gotham Hall, NY. I was able to attend, thanks to an invitation by Marconi Society.  One award in particular resonated with me at a personal level, which was given to Charles Mistretta. He was awarded IEEE Medal for Innovations in Healthcare Technology for the development of techniques that have drastically changed how imaging for cardiovascular diagnosis is done. Were it not for him, and other individuals like him, my father's diagnosis for cardiovascular disease in 1980 may not have been possible, and perhaps I would not have be sitting in the ceremony yesterday.

SPEC Cloud IaaS 2016 Benchmark

I was fortunate to lead the design and implementation of SPEC Cloud IaaS 2016 - the first industry standard benchmark to measure and compare the performance of IaaS Clouds. Read more on my IBM blog entry.

Eclipse Luna on Mac OS X Mavericks

Setting up Eclipse Luna on Mac OS Mavericks is not straightforward. I ran into some issues and had to search bunch of posts to solve the problems.


Here are the steps for getting Eclipse Luna working correctly on Mac OS X Mavericks.

1. Install Java SDK. If you just download JRE, you will run into problems. Eclipse Luna requires java version 7 or higher.
   http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
2. Open a terminal and test that your version of java is correct:
   $ java -version
   java version "1.7.0_79"
   Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
   Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
   
3. Download Eclipse Luna
   http://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/luna/SR2/eclipse-java-luna-SR2-macosx-cocoa-x86_64.tar.gz
4. Copy Eclipse to your Application folder. If you already have an eclipse folder there, make a back up copy.
5. Open Eclipse by double clicking on the Eclipse icon. If you get an error similar to the one below, go to next step
   
   eclipse The JVM shared library "/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/../lib/server/libjvm.dylib" does not contain the JNI_CreateJavaVM symbol.
6. Open the following file
   $ vi /Applications/eclipse/Eclipse.app/Contents/Info.plist
   
   search for Eclipse
   
   Add the following string:
   -vm/Library/Java/JavaVirtualMachines/jdk1.7*.jdk/Contents/Home/bin/java
   
   You can check jdk folder path as follows:
   $ ls /Library/Java/JavaVirtualMachines/ | grep jdk

Cloud SLAs: Present and Future

If your company leverages cloud computing, you should take a look at my recent article, titled, "Cloud SLAs: Present and Future", that appeared in the July 2012 issue of ACM Operating System Review. In this article, I describe SLAs of major cloud providers such as Amazon, Rackspace, Microsoft Azure, Terremark, and Storm on Demand. Key findings are that none of the providers guarantee performance and instead provide some notion of availability. However, the notion of availability varies across cloud providers. EC2, a popular cloud, only guarantees availability on a data center instead of a VM! When you are building an application, it may serve you well to read the SLAs of cloud providers.

I make the job easier for you :). Here is a link to my paper and presentation. I also provide guidance on how SLAs should be designed for future cloud services.

StartupCTO

Startup CTO contains many useful links, amongst them setting up SVN server.

Rethinking cell phone plans

You are a smart phone user with unlimited data, text, and voice post-paid plan. However, you do not use your voice plan much. In fact, your voice plan usage has dropped so much that you will not even fully utilize the lowest tiered voice plan offered by your carrier. Your text usage is also dropping, thanks to applications such as Viber, Skype, and Blackberry messenger. Only your data usage is increasing. However, you are left wondering why do you still have to pay a heft sum for unlimited voice and text plans that you sparingly use?

Ok, ok, carriers (read AT&T and Verizon) do not offer unlimited data plans anymore, unless you were among those lucky customers who purchased smart phones from them a while back (or unless you are a Sprint customer). But wouldn't it be nice if you do not have to pay for voice and text plans per month that you sparingly use?

Sadly, there has been little innovation in cellular data, voice, and text plans which does not benefit the customer at all. The simplest pricing strategy is additive bundling. One must purchase a voice plan, before purchasing a data or text. And all of it from one carrier. Given the uptick in data usage, and given our phones are increasingly sophisticated computers, paying separately for data, voice, and text services, and forcing a user to purchase a voice plan before purchasing a data plan increasing does not make any sense. But what can be done?

Imagine the following plans.

Data only (post or prepaid)
Carriers only offer xG (where x stands for a 3 or 4) data plan, and provide their own free smart phone app for voice calling. This plan can be the cheapest option for users. Data service always works, so that parents can always text their kids. And kids do not pick the phone calls anyway from their parents. Voice calls may work, depending on the signal strength. Call it the teenager plan.

Data (post or prepaid) + voice calls (usage based)
Same as above with a prepaid voice plan. The carrier voice calling app dynamically shifts between xG data plan, WiFi, or a prepaid voice plan depending on the available signal and its strength. The user is only charged for the voice calls it makes. Call it the 35+ plan.

Data (post or prepaid) + voice plan (rollover or unlimited)
Same as above but with a rollover or unlimited voice plan. Call it the everything else, 20-35, or family talk plan.

Data (rollover) + voice plan (rollover)
Data and voice plans roll over from month to month.

Now imagine that you can purchase data + voice plans, but instead you can purchase data from one carrier and voice from another? Why is that useful? Imagine that when you want to upgrade your phone, both your data and voice carriers offer you a phone upgrade option. Wouldn't that be cool?

Such a plan can potentially become a traveler's best friend. For example, when you travel, why do you have to worry about purchasing a separate SIM card for phone and carrying two phones? Just purchase or add credit to a local voice plan app for your smart phone and bingo!

But where is this innovation going to come from? Do not expect to come it from carriers.

The person who may have been able to pull it off in the very near future -- Steve Jobs -- is now dead. Android is all about increasing its installed base, so it will be least likely to upset carriers. Windows phone is too young to force carriers to innovatively price their voice plans.

Some innovation may require regulatory intervention without which one may never be able to purchase a data plan from one carrier and voice plan from other.

Or, perhaps, the innovation in pricing may come from an underdog carrier. Who knows?

Gift registries, groups, and partial payments

I recently made an online purchase for a friend's gift registry on amazon.com and was left wondering about the room for innovation in this space. Although I purchased the gift, but I really wanted to contribute money towards a more expensive item in the gift registry because it is something that my friend will really like. However, the item was too expensive for a single purchase. Unfortunately, there do not seems to be a way to partially pay (or contribute) towards the price of a more expensive gift on Amazon's gift registries. The old fashioned way of doing it is by giving cash as a gift. The online registry creator can also select gift cards of different values which people can purchase. Then there are cash registries where people can 'gift' money for a baby's college fund or honey moon fund, which is just another play on the gift idea cards or vice versa.

But can we do better? Can we leverage the idea of groups, crowd sourcing, and partial payments to innovate in gift registries?

What if there was a Groupon-like way to keep track of partial payments towards gifts? For example, the creator of gift registry can set a limit like, 'if 50% of the costs for a gift are paid by my friends, I will purchase the item'. The person purchasing the gift can have the option of purchasing the gift, contributing towards partial payment of a gift, or simply paying cash. The purchaser can limit the use of cash, i.e., use it only towards gifts etc.

Link plagiarism, Google+, and Facebook

Have you ever seen the link posted by you being re-posted by one of your friends without a 'via', as-if indicating that they were the first to found the link? I call this link plagiarism. Basically, you see a link posted by your friend, and you repost it on your profile without any attribution. Admittedly, I may also be guilty of this -:)

Now, Facebook groups together the same link being posted by your friends and also shows the time they were posted. So you can easily track when did the story broke among your friends and who was the first to report it. I want Google+ to have the same ability. It will be cool to develop a tool that scans one's friends profiles and see who is the habitual plagiarist :)

Friend Recommendation in Google+

When I clicked on 'Find and Invite' in Google+, the results recommended me Larry Page and Sergey Brin among the people I know. Interestingly, the page recommended Larry ahead of Sergey by seven people. (The number of people between Larry and Sergey was higher initially, but it decreased to seven by the time I was writing this post as I added some people as 'Friends'). Obviously, I was intrigued! I have never met any of them before. Moreover, Facebook has never recommended Mark Zuckerberg as a friend to me. So what is going on?

On a bit of digging, I found that one of my Google+ friends has them in their friend's list. So that explains why Google+ is recommending them as the people I know. But why would it rank Larry ahead of Sergey for me? -:) May be, because at this time, there are approximately 12,000 people who have Larry in their circles, but only ~9,000 people who have Sergey in their circles. Surely, Larry is more popular than Sergey :)



But does Google+ only recommends people to me who are friends with my existing friends? The answer is no. Not surprisingly, it seems to use multiple strategies. One strategy is obviously, 'Friends of Friends' for recommendation. That seems to be the easiest approach. The other strategy seems to be a 'Gmail scan'. Without scanning my Gmail, it will never have recommended me people who are not even on Google+. This is an interesting strategy because there are several options on how to implement it. One option is that Google+ scans my inbox for conversations. That is the easiest option. The other option is to scan my inbox for emails from people that are on the same listserv as I am. Finally, the third strategy seems to be a bit of mystery. I will call this the 'Other' strategy. Google+ recommended me people that are not 'Friends of Friends', and with whom I have not exchanged an email and I think that we are not in the same listserv.

vSphere snapshot deletion 95% hang

Recently, I was trying to delete snapshots on ESXi and ran into some problems. When I deleted the snapshot, vSphere snapshot removal was stuck on 95%. I found some nice articles from VMware on how to resolve this, such as (1), (2), (3).

I was starting to get errors such as: "unable to connect to the mks: error connecting to /bin/vmx process" and the VM will not start.

I decided to delete another VM on the same PM in order to free up space for the VM stuck with 95% snapshot completion. That helped resolve the problem.

datemyschool.com -- an analysis

Recently, a friend of mine pointed me to datemyschool.com, a website that facilitates meeting and dating people from different schools within a person's enrolled university and some other universities. The main idea is that a person signs up with their university email address, and indicates their preference in a person of opposite sex, same sex, or both at the sign up time. (The preference can also be changed later through website settings.) Once the user logs in, he or she can view the profiles of other people from the different schools within the same university, or other universities. For example, people from Columbia, NYU, FIT, Princeton, and Yale can view profiles of each other. (Initially, the site was restricted to Columbia, NYU, and FIT).

The website is very interesting from the perspective of a social science researcher as it provides a useful window into the relationship dynamics of students in different schools and universities, very easily.

I computed some statistics on the number of people looking for a relationship or friendship within different schools at Columbia and NYU. There statistics were computed on March 19th and 20th, 2011. The results are shown in the bar graphs below. Each graph displays the number of females and males according to a certain metric. The number of females and males are computed using the 'Calculate' button in the 'Search' page of datemyschool.com . The number of males include those who are looking for females or looking for both males and females, and vice versa. The website does not allow restricting search to people who are only looking for friendship or relationship. The website also does not allow searching for people who are alumni or are currently enrolled.

Most of the results may not be considered surprising, but are definitely interesting.

The salient features of these statistics are:

• There are approximately 3800 users in both Columbia and NYU (Figure 1). The site is more popular at Columbia than NYU.
  
• The ratio of males and females within Columbia and NYU is comparable, but the ratio varies greatly with different schools in each university.
  
• The number of people with a profile picture is approximately 50-60% of the total population (Figure 2). These people may be considered active users.
  
• People with a profile picture almost always specify their ethnicity (Figure 4).
• However, only 50-60% of people with a profile picture specify their religious preference, even though religious preference contains options such as 'Other' (Figure 3).
• Graduate students are more likely to upload a picture than undergraduate students. Approximately, one third on undergraduates upload their picture instead of 50% for graduate students.
  

Some Columbia specific salient insights are:

• Business, engineering, and GSAS have more males than females, whereas teachers college, social work, and medical center have the opposite trend. The distribution of males and females in these schools confirms the commonly held stereotypes. For example, there are many engineering make students looking for females (155 vs 16).
• More men declare themselves to be atheist than woman (54 vs 29) where as more woman declare themselves to be spiritual than men (63 vs 48)
• People with agnostic, atheist, or spiritual beliefs approximately comprise 50% of the total who indicate their religious preference.
• There are more older males than females.
  

These results can be interpreted in many different ways and can also be used to compute meeting probabilities. However, I have left these interpretations and computations for the reader.



Figure 1: Number of females and males in Columbia and NYU with a profile.



Figure 2: Number of females and males in Columbia and NYU that have a picture with a profile. Observe that only 50% of the people upload a profile picture.



Figure 3: Number of Columbia and NYU students with a profile pic, sorted according to their religion preference. It is optional to specify any of the religion choices as shown in the figure above.



Figure 4: Number of Columbia and NYU students with a profile pic, sorted according to their ethnicity preference. It is optional to specify any of the ethnicity choices as shown in the figure above.

Observe from Figure 3 and 4 that of all the Columbia and NYU students with a profile pic, only 52%-63% specify their religious preference, while more than 90% specify their ethnicity preference.




Figure 5: Of the 311 and 305 Columbia female and male students who have a profile pic and specify their religious preference, the above figure shows the number of people who have selected a particular category.



Figure 6: Of the 560 and 582 Columbia female and male students who have a profile pic and specify their ethnicity preference, the above figure shows the number of people who have selected a particular category.



Figure 7: Age distribution of females and males at Columbia, including the ones with a profile picture. The total numbers do not add up to the numbers in Figure 1 and 2, but I am not sure of the reason.


Figure 8: The total number of male and female students in different Columbia schools. The figure also shows the number of males and females who have uploaded a picture.

(The views expressed in this article are my own and do not reflect the views of my employers)

Connecting iPad with EAP-TLS security

Recently, a friend of mine asked me to connect his iPad to a network that requires EAP-TTLS security. By default, iPad does not have EAP-TTLS profile installed. To install this profile on iPad, I downloaded the iPhone configuration utility . Then, I setup the configuration by selecting the EAP profile and selecting TTLS checkbox. Then, I selected the PAP security from the inner security dropdown box since my friend's network requires it. Here is a snapshot.



Then, I selected the WPA / WPA2 Enterprise. entered the name of the network (not shown below) and entered the user name and password.



Enter your profile name and identity as shown below. The example is for Columbia medical center.



Then, I installed the profile on iPad. The figure below shows the profile that has been installed.



Your should now be connected to your network.

Skype-Facebook partnership: A response to Google (Talk+email)?

http://www.cnn.com/2010/TECH/social.media/10/04/facebook.skype.partnership/index.html

Social media meets real-time communication? Facebook needs Skype because they do not have a real-time communication network. Skype needs Facebook because they do not have access to their user's email graph (like Google). Its really Facebook and Skype's answer to Google's growing threat to their businesses.

Using tc to regulate downlink speed

tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate
100kbit burst 10k drop flowid :1

The second command limits the download rate to 100kbit by dropping packets. Since we do not want to drop too many packets, we setup a burst size size of 10k.

See this link for more details.
http://lartc.org/howto/lartc.cookbook.ultimate-tc.html

Using tc to regulate uplink speed

Here is a nice script that I found:

----------------

#!/bin/bash
#
# tc uses the following units when passed as a parameter.
# kbps: Kilobytes per second
# mbps: Megabytes per second
# kbit: Kilobits per second
# mbit: Megabits per second
# bps: Bytes per second
# Amounts of data can be specified in:
# kb or k: Kilobytes
# mb or m: Megabytes
# mbit: Megabits
# kbit: Kilobits
# To get the byte figure from bits, divide the number by 8 bit
#

#
#Name of the traffic control command.
TC=/sbin/tc

# The network interface we're planning on limiting bandwidth.
IF=eth0 # Interface

# Download limit (in mega bits)
DNLD=1mbit # DOWNLOAD Limit

# Upload limit (in mega bits)
UPLD=1mbit # UPLOAD Limit

# IP address of the machine we are controlling
IP=128.59.9.113 # Host IP

# Filter options for limiting the intended interface.
U32="$TC filter add dev $IF protocol ip parent 1:0 prio 1 u32"

start() {

# We'll use Hierarchical Token Bucket (HTB) to shape bandwidth.
# For detailed configuration options, please consult Linux man
# page.

$TC qdisc add dev $IF root handle 1: htb default 30
$TC class add dev $IF parent 1: classid 1:1 htb rate $DNLD
$TC class add dev $IF parent 1: classid 1:2 htb rate $UPLD
$U32 match ip dst $IP/32 flowid 1:1
$U32 match ip src $IP/32 flowid 1:2

# The first line creates the root qdisc, and the next two lines
# create two child qdisc that are to be used to shape download
# and upload bandwidth.
#
# The 4th and 5th line creates the filter to match the interface.
# The 'dst' IP address is used to limit download speed, and the
# 'src' IP address is used to limit upload speed.

}

stop() {

# Stop the bandwidth shaping.
$TC qdisc del dev $IF root

}

restart() {

# Self-explanatory.
stop
sleep 1
start

}

show() {

# Display status of traffic control status.
$TC -s qdisc ls dev $IF

}

case "$1" in

start)

echo -n "Starting bandwidth shaping: "
start
echo "done"
;;

stop)

echo -n "Stopping bandwidth shaping: "
stop
echo "done"
;;

restart)

echo -n "Restarting bandwidth shaping: "
restart
echo "done"
;;

show)

echo "Bandwidth shaping status for $IF:"
show
echo ""
;;

*)

pwd=$(pwd)
echo "Usage: tc.bash {start|stop|restart|show}"
;;

esac exit 0

Limiting bandwidth per process in Linux

A friend of mine recently asked me how to limit bandwidth per process in Linux. Here are the steps for accomplishing this using iptables and tc. The idea is to create a traffic class, specify the class priority and the bit-rate etc, tell the traffic class that packets with a handle [x] should be handled by this class, and assign packets a handle using iptables. iptables allows to assign a handle using gid or pid. Here are the commands. These commands do not limit incoming traffic so, one has to set iptables for INPUT accordingly.

#setting the root queuing class (can be ignored)
tc qdisc add dev eth0 root handle 1: htb default 15

#specify the class with priorty 1. The class id is 1:1
tc class add dev eth0 parent 1: classid 1:1 htb rate 10kbit ceil 10kbit prio 1

#specify the filter. so packets with handle '10' go into the class 1:1
tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 10 fw classid 1:1

#now give the packets a handle of 10
iptables -t mangle -A OUTPUT -m owner --gid-owner [gid] -j MARK --set-mark 0xa

A good description of of traffic shaping in Linux can be found here.
http://lartc.org/howto/lartc.cookbook.fullnat.intro.html

Matlab and embedding fonts in pdf

I typically create figures using Matlab and incorporate the eps files in my pdf file. Matlab print function does not correctly embed the fonts in the pdf file. However, there are many ways of generating eps files in Matlab with correctly embedded fonts:
http://www.mathworks.com/matlabcentral/newsreader/view_thread/172388

Here is another link that I found useful:
http://do.whileloop.org/soft/tricks/pdflatex-fonts.php