Imagine you are a security manager being asked to do a security assessment on a new software for your organisation. It will be deployed across all Windows workstations and servers and operate as a boot-start driver in kernel mode, granting it extensive access to the system. The driver has been signed by Microsoft’s Windows Hardware Quality Labs (WHQL), so it is considered robust and trustworthy. However, additional components that the driver will use are not included in the certification process. These components are updates that will be regularly downloaded from the internet. As a security manager, would you have any concerns?
I would be, but what if it were a leading global cybersecurity vendor? Do we have too much assumed and transitive trust in cybersecurity vendors?
The recent CrowdStrike Blue Screen of Death (BSOD) incident has raised significant concerns about the security and reliability of kernel-mode software, even when certified by trusted authorities. On July 19, 2024, a faulty update from CrowdStrike, a widely used cybersecurity provider, caused thousands of Windows machines worldwide to experience BSOD errors, affecting banks, airlines, TV broadcasters, and numerous other enterprises.
This incident highlights a critical issue that security managers must consider when assessing new software, particularly those operating in kernel mode. CrowdStrike’s Falcon sensor, while signed by Microsoft’s Windows Hardware Quality Labs (WHQL) as robust and trustworthy, includes components that are downloaded from the internet and not part of the WHQL certification process.
The CrowdStrike software operates as a boot-start driver in kernel mode, granting it extensive system access. It relies on externally downloaded updates to maintain quick turnaround times for malware definition updates. While the exact nature of these update files is unclear, they could potentially contain executable code for the driver or merely malware definition files. If these updates include executable code, it means unsigned code of unknown origin is running with full kernel-mode privileges, posing a significant security risk.
The recent BSOD incident suggests that the CrowdStrike driver may lack adequate resilience, with insufficient error checking and parameter validation. This became evident when a faulty update caused widespread system crashes, indicating that the software’s error handling mechanisms could not prevent catastrophic failures.
For security managers, this incident serves as a stark reminder of the potential risks associated with kernel-mode software, even when it comes from reputable sources. It underscores the need for thorough assessments of such software, paying particular attention to:
1. Update mechanisms and their security implications
2. The scope of WHQL certification and what it does and does not cover
3. Error handling and system stability safeguards
4. The potential impact of software failures on critical systems
While CrowdStrike has since addressed the issue and provided fixes, the incident has caused significant disruptions across various sectors. It has also prompted discussions about balancing rapid threat response capabilities and system stability in cybersecurity solutions.
In conclusion, this event emphasises the importance of rigorous security assessments for kernel-mode software, regardless of its certifications or reputation. Security managers must carefully weigh the benefits of such software against the potential risks they introduce to system stability and security.
Agile software development methodologies have become popular in the last decade or so. One of such methodologies, DevOps, has established itself among the most used methodologies across various industries throughout the globe. However, for security teams, this popularity of DevOps has increased attack surface, which is continuously changing that requires protection. This post will focus on how we got here and why DevOps developers are becoming the bad actor’s favourite target.
The last two decades have seen a tremendous change in how companies have adopted new ways of doing business. Digital transformation helped businesses innovate, bringing new products and offerings for their customers. However, since every business is doing the same, time to market has gone down, and businesses need to keep innovating at speed and scale. DevOps helped application delivery to meet the business requirement.
DevOps teams are usually part of the business and always looking for quick innovations that leave them vulnerable. Business loves this flexibility and not being part of the traditional technology or IT teams. Since the DevOps teams are often part of the business, they give into the business demands, especially during the times like today when COVID19 has presented unseen business challenges. This meant the rigour and the traditional development process were either not properly followed or were deliberately ignored in the name of need and speed.
Companies need for speed and innovation often leads teams to use tools and techniques to create new DevOps pipelines, and in doing so, many new threat vectors are introduced, giving hackers their target. Hackers are very much aware of developer psychology, where they use third-party tools and libraries to shorten the development lifecycle. Today, everything is a code, and DevOps teams are pursued by the threat actors targeting cloud credentials stored in repositories like GitHub.
In the same way, the source code, usually a business IP(intellectual property), is targeted to gain direct access to the IT infrastructure and data. Hackers often leverage outdated third-party libraries in the build pipelines to inject malicious code before the wider deployment of the code as a primary or a secondary attack. Supply chain attacks are a classic example of such breaches. The recent SolarWind and Kaseya attacks were executed by inserting malicious code in their products.
Every stage of the DevOps process is vulnerable to the threats of malicious activity leading to data loss for the business, which may also impact the downstream customers of the company.
As mentioned above, due to how DevOps teams are structured, segregation of duties is often ignored, making them the most privileged users in the business having access to production and non-production environments. Therefore, if developer credentials are compromised by a well-crafted phishing attack or by other means such as machine compromise, then it is not only the sources code is at risk but also the risk of losing the customer data.
We have often seen that business demands excessive access for their DevOps developers. They are often over-provisioned with systems access in the name of speed and the anticipation that the developer might need to do their jobs in the future. For sure, this improves developer efficiency, but all the security principles regarding access controls learnt and developed over the years go down the drain. As noted above in the post, if a hacker sets his hand on such an over-provisioned developer credentials, it can create havoc by inserting malicious code into the application, leaking source code, changing security control, and the most pervasive of all is by stealing production customer data.
Due to this very reason, developer credentials have become an attractive target for hackers. Therefore, organisations must take adequate steps to protect developer credentials as “crown jewels”.
It isn’t easy to get this right unless the business and the security organisation create an integrated DevOps where security is inherently integrated with the development process.
The following are some of the recommendations to start this journey.
* Least-privilege policy: Implementing a least privileges policy is the only real deal without which it is only a matter of time that you will be breached. Traditionally, user access is only reviewed for the end-users and not the developer’s access. Businesses must implement user access reviews for developer access so that access privileges are always aligned to their roles in performing their jobs. This will help in limiting exposure to credential breaches.
* Make it hard: Implement strong authentication practices for privileged users. Implement multi-factor authentication, single sign-on and IP safe listing practices to help reduce the risk of developer credentials falling into the hands of hackers.
* There is no place for credentials in the code: Whether production or non-production environments, storing secrets in the code is a “bad idea”. Code must be checked for stored secrets, especially in production environments. Automated tools such as AWS Secrets Manager or CredStash be used for managing developer credential management.
* Automate developer access management: The developer accounts are often kept outside the automated user access provisioning and de-provisioning using single sign-on or linked with HR systems. This leads to poor management of developer access, failing to revoke their access on role change or leaving the company.
* Shift left (Automate, automate, automate…): Embed security processes in DevOps pipelines to ensure security controls are embedded right in the development process. Seek opportunities to automate these processes to reduce human intervention as much as possible. For example, no code is merged into the master branch without going through the review process, checking for a series of security tests that include secret detection and peer review. Enforce automated policies to stop code from being pushed without these tests. Repositories are rigorously monitored for changes, so malicious actors cannot circumvent the security tests.
* Audit, audit and audit: Ensure systems are not provisioned with default credentials. Periodically audit production systems for default or embedded credentials so that the credentials are not left in the code, helping hackers to have a field day.
* Train business and DevOps developers: To err is human, and developers are humans bound to make mistakes. There is no substitute for training, and organisations must spend energy to equally train their Developers and business. Security teams must develop security champions and DevOps teams to promote and implement good security practices.
It is important to focus on better protection of the DevOps environment. Threat actors, including nation-states, will continue to find new ways to breach your defences. If businesses fail to protect DevOps teams, they will continue targeting developers as a way of gaining access to “crown jewels”.
Globally, businesses leveraged the benefits of transforming their businesses by adopting new ways of doing business and delivering their products to the market quickly and efficiently. The digital transformation has made a distinctive contribution to this effort. Organisations use modern and efficient applications to deliver the above business outcomes. APIs, behind the scenes, are the most critical components that help web and mobile applications deliver innovative products and services.
API is a piece of software that has direct access to the upstream or downstream applications and, in some cases, directly to the data. The following picture depicts a typical scenario where a web application calls an API, which calls downstream resources and data. Unfortunately, due to the nature of the API, direct access to the data introduces a new attack surface called “API Breaches”, which is continuously on the rise, resulting in impersonation, data theft, and financial fraud.
According to Gartner, by 2022, the outlook of API breaches will change from an infrequent breach to the most-frequent attack vector, which will result in frequent data losses for enterprise web applications. This changing trend has brought the realisation that something needs to be done to protect data at the API and the digital interfaces level.
Example of an API Breach
There are many factors of API breaches, and one of the common issues is the broad permissions within the application ecosystem intended for a non-human interaction. Stricter controls regarding user access ensure only the authorised user can access the application. Once a user is authenticated, the APIs can frequently access any data. This is all said and done, but the problem starts when a bad actor bypasses the user authentication accessing the data from the downstream systems.
The above picture shows a user accessing a mobile application using an API while the desktop users connect the same API accessing the same backend through a web interface. Mobile API calls include URIs, methods, headers, and other parameters like a basic web request. Therefore, the API calls are exposed to similar web attacks, such as injection, credential brute force, parameter tampering, and session snooping. Hackers can employ the same tactics that would work against a traditional web application, such as brute force or injection attacks, to breach the mobile API. It is inherently difficult to secure mobile applications. Therefore, attackers are continuously decompiling and reverse engineering mobile applications in pursuit of finding vulnerabilities or opportunities to discover hardcoded credentials or weak access control methods.
API Security Threats
OWASP.ORG, in 2019 released a list of top10 API Security Threats advising on the strategies and solutions to mitigate the unique security challenges and security risks of APIs. Following are the ten API security threats.
API1:2019: Broken Object-Level Authorisation
APIs often expose endpoints handling object identifiers. This exposure leads to a wide attack surface where a user function used to access a data source creates a Level Access Control issue. Therefore, it is recommended to carry out object-level authorisation checks for all such functions accessing data.
API2:2019: Broken User Authentication
User Authentication is the first control gate accessing the application, and attackers often take advantage of the incorrectly applied authentication mechanisms. It is found that the attackers may compromise an authentication token or exploit flaws assume other user’s identity temporarily or permanently. Once compromised, the overall security of the APIs is also compromised.
API3:2019: Excessive Data Exposure
In the name of simplicity, developers often expose more data than required relying on the client-side capabilities to filter the data before displaying it to the user. This is a serious data exposure issue and therefore recommended to filter data on the server-side, exposing only the relevant data to the user.
API4:2019: Lack of Resources and Rate Limiting
System resources are not infinite, and often poorly designed APIs don’t restrict the number or size of resources. Excessive use of the resources may result in performance degradation and at times, can cause Denial of Service (DoS), including malicious DoS attacks. Without rate limiting applied, the APIs are also exposed to authentication vulnerabilities, including brute force attacks.
API5:2019: Broken Function-Level Authorisation
Authorisation flaws are far more common and are often the result of overly complex access control policies. Privileged access controls are poorly defined, and unclear boundaries between regular and administrative functions expose unintended functional flaws. Attackers can exploit these flaws by gaining access to a resource or enacting privileged administrative functions.
API6:2019: Mass Assignment
In a typical web application, users update data to a data model often bound with the data that users can’t. An API endpoint is vulnerable if it automatically converts client parameters into internal object properties without considering the sensitivity and the exposure level of these properties. This could allow an attacker to update object properties that they should not have access to.
API7:2019: Security Misconfiguration
Security misconfiguration is a very common and most pervasive security problem. Often inadequate defaults, ad-hoc or incomplete configurations, misconfigured HTTP headers or inappropriate HTTP methods, insufficiently restrictive Cross-Origin Resource Sharing (CORS), open cloud storage, or error messages that contain sensitive information leave systems with vulnerabilities exposing to data theft and financial loss.
API8:2019: Injection
When untrusted data is sent to the interpreter alongside the command or query results in Injection flaws (including SQL injection, NoSQL injection, and command injection). Attackers can send malicious data to trick the interpreter into executing commands that allow attackers to access data without proper authorisation.
API9:2019: Improper Asset Management
Due to the nature of the APIs, many endpoints may be exposed in modern applications. Lack of up-to-date documentation may lead to the use of the older APIs, increasing the attack surface. It is recommended to maintain an inventory of proper hosts and deployed API versions.
API10:2019: Insufficient Logging and Monitoring
Attackers can take advantage of insufficient logging and monitoring, coupled with ineffective or lack of incident response integration, to persist in a system to extract or destroy data without being detected.
Common Attacks Against APIs
APIs, similar to traditional applications, are also exposed to many of the same types of attacks that we defended to protect networks and web applications. Following are some of the attacks that can easily be used against APIs.
How to Secure APIs
Some other broader controls that organisations may implement to protect their publicly shared APIs in addition to the mitigation strategies mentioned in the above table.
“Security First”, prioritise security.
Often security is an afterthought and seen as someone else’s problem, and API security is no different. API security must not be an afterthought. Organisations will lose a lot with unsecured APIs. Therefore, ensure security remains a priority for the organisation and built into your software development lifecycle.
Maintain a comprehensive inventory of APIs.
It is so commonly seen that several APIs are publicly shared, and often, organisations may not be aware of all of them. To secure these APIs, the organisation must first be aware of them. A regular scan shall discover and inventory APIs. Consider implementing an API gateway for strict governance and management of the APIs.
Use a strong authentication and authorisation solution.
Poor or non-existent authentication and authorisation controls are major issues with publicly shared APIs. APIs are designed not to enforce authentication. It is often the case with private APIs, which are only for internal use. Since APIs provide access to organisations’ databases, the organisations must employ strict access controls to these APIs.
Least privileges.
The foundational security principle of “least privileges” holds good for API security. All users, processes, programs, systems and devices must only be granted minimum access to complete a stated function.
Encrypt traffic in transit.
API payload data must be encrypted when APIs exchange sensitive data such as login credentials, credit card, social security, banking information, health information etc. Therefore, TLS encryption should be considered a must.
Limit data exposure.
Ensure development data such as development keys, passwords, and other information that must be removed before APIs are made publicly available. Organisations must use scanning tools in their DevSecOps processes, limiting accidental exposure of credentials.
Input validation.
Input validation must be implemented and never be sent through the API to the endpoint.
Use rate limiting.
Setting a threshold above which subsequent requests will be rejected (for example, 10,000 requests per day per account) can prevent denial-of-service attacks.
Deploy web application firewall.
Deploy a web application and configure it to ensure that it can understand API payloads.
Conclusion
APIs are the most efficient method of accessing data for modern applications. Mobile applications and Internet of Things(IoT) devices leverage this efficiency to launch innovative products and services. With such a dependency on APIs, some organisations may not have realised the API specific risks. However, most organisations already have controls to combat well-known attacks like cross-site scripting, injection, distributed denial-of-service, and others that can target APIs. In addition, the best practices mentioned above are also likely practised in these organisations. If you are struggling to start or know where to start, the best approach could be to start from the top and start making your way down the stack. It doesn’t matter how many APIs your organisation chooses to share publicly. The ultimate goal should be to establish principle-based comprehensive and efficient API security policies and manage them proactively over time.
I did my FAIR analysis fundamentals course a few years ago and here are my thoughts on it.
FAIR stands for Factor Analysis of Information Risk, and is the only international standard quantitative model for information security and operational risk. (https://www.fairinstitute.org/)
My interest to learn more about FAIR came from two observations.
The first was that we had many definitions of what constitute risk. We refer to “script-kiddies”as risks. Not having a security control is referred to as risk. SQL injection is a risk. We also said things like “How much risk is there with this risk?”
The other observation was with our approach at quantifying risk. We derived the level of risk based on the likelihood and impact. And sometimes it was hard to get agreement on those values.
Having completed the course, one of the things I like about FAIR is their definitions. Their definitions of what is a risk, and what it must included. It should include an asset, threat, effect with a method that could be optional. An example of a risk is the probability of malicious internal users impacting the availability of our customer booking system via denial of service.
It uses future loss as the unit of measurement rather than a rating of critical, high, medium & low. The value of future loss is expressed as a range with a most likely value along with the confidence level of that most likely value. As such it focuses on accuracy rather than precision. I quite like that as it makes risk easier to understand and compare. Reporting that a risk has a 1 in 2 year probability of happening with a loss between $20K to $50K but likely being $30K is a lengthy statement. However it is more tangible and makes more sense than reporting that the risk is a High Risk.
Now it sounds like I’m all for FAIR, but I have some reservations. The main one being that there isn’t always data available to determine such an empirical result. Risk according to FAIR is calculated by a multiplication of loss frequency (the number of times a loss event will occur in a year) with loss magnitude (the $ range of loss from productivity, replacement, response, compliance and reputation). It’ll be hard to come up with a loss frequency value when there is no past data to base it on. I’ll be guessing the value and not estimating it. FAIR suggests doing an estimate for a subgroup if there isn’t enough reliable data available, but again I see the same problem. The subgroup for loss frequency is the multiplication of number of time the threat actors attempt to effect the asset with the percentage of attempts being successful. Unless you have that data, that to me is no less easier to determine.
Overall it still feels like a much better way of quantifying risk. I’ll end with a quote from the instructor. “Risk statements should be of probability, not of predictions or what’s possible.” It resonated with me as it is something I too often forget.
Edwin Kwan
