Thursday, April 10, 2014

Beware: the HeartBleed Bug

One of the most scariest bugs (CVE-2014-0160) has been recently discovered in the OpenSSL cryptographic library. From heartbleed.com:
The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.
The Eucalyptus Systems has released a security advisory about EuStore images that are potentially affected by this vulnerability. The advisory can be found at:

Other Eucalyptus products also rely on OpenSSL library as a dependency and potentially can be affected by the HeartBleed Bug if a vulnerable version of OpenSSL is installed on the host. We encourage all Eucalyptus users to follow the best practices of system hardening and to ensure that their Eucalyptus installs have been upgraded to include the latest OpenSSL packages:

For more information on affected versions, refer to the official advisories for the supported Linux distributions:



Friday, February 14, 2014

The Art of Security Code Reviews

Reviewing code for security is one of the important building blocks of ensuring software security. I consider code analysis, in general, to be a very powerful and universal way to get the most detailed information about a piece of software. It can be used in different situations, from catching bugs in a newly developed feature to deriving a security model of a third party application (if code is available, of course). In my work, I dedicate a considerable amount of time to reviewing code for security. I find that, while the purpose and scope of reviews differ depending on a situation, the general approach that I use to decompose and analyze code for security often stays the same. Here is a list of high-level things that I usually look for:
  • Application entry and exit points -- understanding all possible intended and unintended ways data enters and exits the application (e.g., through APIs, user interfaces, and filesystem access) is the foundation for any further analysis
  • Control and data flow -- also, intended and unintended 
  • Trust boundaries and their crossings 
  • Authentication and authorization enforcement on all entry points and trust boundary crossings
  • Input validation on all entry points and output escaping when needed
  • Protection of data at rest and in flight
  • Permissions (filesystem, processes, etc)
  • Injection points -- any change of context in which data is used creates possibilities for injections
  • Any explicit and implicit assumptions done by the code and ways to violate them
  • Any application domain-specific bad practices and vulnerabilities (such as CSRF vulnerabilities in web applications) -- this can be an art on its own and requires good understanding of the domain and known problems
  • Any language or technology specific bad practices
To achieve better precision of the analysis, I recommend to combine code analysis with white box testing. Targeted testing that is guided by information gathered through code review is very helpful in, for example, navigating through complex control flow or validating problem areas.

Use of static analysis tools (if such are available for the target language) is another important component of doing a thorough code analysis. Static analysis tools usually achieve high and systematic coverage of all possible execution paths that is hard to ensure with human analysis. They range from tools looking for common known-to-be-bad patterns in the code to more sophisticated tools doing more context-sensitive analysis to identify more complex issues. Yet, it's important to understand that static analysis tools should be used as a complement, not a replacement, to human analysis because they suffer from various inherent limitations (which are outside the scope of this post) that allow them to discover only a subset of bugs that can be discovered with diligent human analysis. For example, identifying bugs that require understanding of program semantics or intended behavior generally requires human involvement.

If done properly and thoroughly, code reviews can be very time-consuming and tedious. However, they should never be skipped regardless of all other activities employed around software security (such as specification/design reviews, penetration testing, and education of developers). At the end of the day, security is always about details (it's about one thing gone wrong rather than everything done right) and code is a good place to discover some of the most subtle ones. I would even argue that given enough scrutiny, time, and human resources, most bugs in the system could be discovered just by looking at its code. In real life, of course, code reviews are time and effort bounded and finding the most potentially severe bugs in a reasonable amount of time is an art, requiring good understanding of the application's domain and its threat model.

Wednesday, April 10, 2013

An Iceberg of Software Security


What is software security? What makes one software product more secure than another? Is it a set of security features that it supports? Or is it a number of security tests that it passes?  Or maybe it's a set of best practices that it follows? The answer is none of the above and all of them together plus more. For example, is a product that implements a two-factor authentication more secure than the one that only relies on one factor? In theory, yes, but in practice it depends on the correctness of the implementation of each feature. A security feature without a proper design, implementation, testing, and correct usage does not automatically make a product secure. True software security is much more complex than a set of features, tests, or documents. It is defined by a complex interaction of multiple factors, with every single one of them being essential. At the end, security is usually measured not by how much of it was done correctly, but by how much was actually missed and compromised as a result.

That's why software security is hard and requires a systematic approach with continuous execution. In my experience in working on software security, I find that different groups of people usually care about or emphasize different aspects of software security. Depending on a situation, prioritizing one security aspect over another might make sense, but in order to evaluate an overall security of a product, generally, they all need to be considered together. More specifically, I believe that there are four main areas that need to be addressed in order to build secure software that can also can be deployed and used securely:
  1. a security stance
  2. secure software 
  3. security features
  4. security documentation 
Having a security stance means to have a clear story about a product's security architecture and risks, security controls, hardening and limitations. A well-defined security stance helps maintaining and improving security of the product as new features are implemented and deployed. To build secure software, i.e. software that can withstand malicious attacks, security should be built in into its development lifecycle. More specifically, there should be a continuous process of evaluating and managing security risks (new and old), taking them into consideration and addressing in the architecture, design, implementation, testing, and deployment of software. It's important to note that this applies not only to new feature or functionality, but also to the entire product as any addition has an associated security risk and potential consequences on the overall security of the system. Building security into software development lifecycle is a complex and perhaps the most time consuming component of software security as it's the only way to ensure that everything else (i.e., the security stance, features and documentation) holds.

To use the iceberg analogy, the security stance and how security of a software is ensured are the fundamental parts of security but they are are mostly invisible to end users. In fact, I'd speculate that they are not that much appreciated and often overlooked by a general public. They're often assumed to be given until something goes wrong. What users usually see and appreciate is features and functionality. Together with documentation they make up the tip of the iceberg. While security features are obviously necessary, they should not be mistaken for software security. It's not a number of features that matters, but correctness of their design and implementation and their contribution to overall security of the product.  Each piece of software should have a set of fundamental security features, such as authentication and authorization, and a set of features that are necessary for its secure usage and deployment given its purpose and design.

Documentation is another important aspect of software security. Providing an end user with a clear explanation of the supported security controls and features and architectural constraints is truly essential to the security of the final deployed product. Documentation helps users to evaluate, understand, correctly use and improve on available security controls. In fact, lack of proper documentation can hurt security of a system if less secure options are chosen due to lack of user understanding of available controls.

As a part of security initiative in Eucalyptus Systems, we are working on addressing all four areas explained above. Putting it all together is not easy and does not happen overnight, but given enough determination it's absolutely possible. In the last few months, we have been mainly focusing on reviewing and defining Eucalyptus security architecture (more on that topic in one of my next posts) and incorporating security concerns into the product development lifecycle (see my  previous post). Eucalyptus user-facing security features so far have been mainly defined by AWS features, such as IAM and EC2 security groups, but this most likely be changing as differences between private and public cloud will become more prominent. Security-related documentation is the area where Eucalyptus probably lacks the most and needs to improve on. Eucalyptus Installation Guide already contains some information related to security, but this information needs to be presented in a more concise and uniform way. To start off, in the upcoming (3.3) release, we will improve our documentation around how to setup firewalls around Eucalyptus components. For the following release, a Eucalyptus hardening guide and IAM best practices are some of the high priority items. 











Friday, March 1, 2013

Building Security In


In a cloud environment, where resource virtualization and multi-tenancy are some of the key features, security is something that simply cannot be ignored. In Eucalyptus, we invest a lot of effort into building software that is not only robust and easy to use, but is also secure and allows for building secure clouds. We also believe that the most effective way of building secure software is to be proactive about it and address security concerns before they become vulnerabilities.

There is no such thing as perfect security, just like quality and robustness, software security is something that has to be worked on continuously and systematically. Even if there are no ways to get to a perfection, a desired level of security can be achieved through careful risk management. Bruce Schneier said in his famous essay that "security is a process, not a product." Following this school of thought and the best practices of building security into software, in Eucalyptus, we started taking security consideration into account throughout the software development lifecycle. We are working on incorporating threat modeling and risk analysis, design and code reviews, static analysis scans, and security testing into the development process.

Eucalyptus also follows an open source development model and its source code and design docs are widely available. We believe that integrating closer with the open source community and having more people activity engaged with our software will help us in multiple areas of software development, including security. We encourage our community to pay close attention to security aspects in Eucalyptus.

No security process is complete without a process for managing security bugs, especially in the open source/development environment when bugs and fixes are generally publicly open. Security issues can be reported to the Eucalyptus Security Team following instructions on the security procedures page. All new security issues, reported both internally and externally, are assessed, assigned a risk score, and prioritized. Security issues that are classified as vulnerabilities are carefully managed internally to prevent premature disclosure of information before fixes are made available to current users. Also, to protect our users running supported version of Eucalyptus from potential attacks and give them time to upgrade, all vulnerabilities are first fixed in maintenance releases and details of a vulnerability and sources of the fix are not made public until about 2 weeks after the software becomes available. At the time of the public announcement, a Eucalyptus Security Advisory (ESA) is published on the Eucalyptus advisories page for each vulnerability.  All vulnerabilities are also assigned a CVE score and published on the Common Vulnerabilities and Exposures website.