Introducing CVE Markdown Charts - Part 1

Posted Mar 19, 2022 Updated Sep 30, 2022

By clearbluejar 18 min read

TL;DR - CVE Markdown Charts - Your InfoSec reports will now write themselves… After writing several InfoSec reports and researching CVEs, I discovered a means to create dynamic charts that help readers and myself understand various CVE relationships and their implications.

Say hello to CVE Markdown Charts, or at least its first iteration (v0.1.0). CVE, as in Common Vulnerabilities and Exposures, where a CVE ID (something in the form of CVE-2022-1234) references a specific instance of a known vulnerability in software. Markdown, as in the popular writing markup language used everywhere by bloggers and developers alike. Charts, as in well, charts, but generated with Mermaid.js.

This post is the first part of what will probably become a short series of posts sharing lessons learned during the development of CVE Markdown Charts. This first one is about that 80% solution. And the sequel, coming later, will describe that last 20% that will take me 3x as long to complete.

We begin with the reason for the project, and the steps taken to solve the issue, and show an initial proof of concept.

The Issue

Typically, when I study a new software component as related to security, or perhaps for InfoSec writing or research, I start by reviewing the CVEs for said component to get a basic understanding of the security issues and common bugs for the software. To find related CVEs, I search for them on cve.mitre.org using a keyword like “Windows Print Spooler” or “Google Chrome Use After Free”. The results from the search will come back in a table with matches from the CVE ID directly or from keywords in the CVE description.

Windows Print Spooler Search Results -cve.mitre.org

MSRC API

Besides Mitre, I have also been using the Microsoft Security Response Center (MSRC) security updates when researching Windows CVEs. Each security update, formatted in the Common Vulnerability Reporting Framework (CVRF) JSON format, contains details about the latest CVEs (release date, links to patches, products affected, etc.). The monthly updates each have a unique CVRF ID like 2021-Dec. MSRC makes the updates available via their MSRC API, which you can test out using their swagger front end. I had been using their API to download the relevant security update JSON pulling out detailed information for any Microsoft CVEs. One of the MSRC REST endpoints provides a convenient CVE ID -> CVRF ID map.

Updates Endpoint

A query for CVE-2020-1048:

  
% curl -X GET --header 'Accept: application/json' "https://api.msrc.microsoft.com/cvrf/v2.0/Updates('CVE-2020-1048')"

The response:

  
{
    "@odata.context": "https://msrc-cvrf.azurewebsites.net/odata/$metadata#Updates",
    "value": [
        {
            "ID": "2020-May",
            "Alias": "2020-May",
            "DocumentTitle": "May 2020 Security Updates",
            "Severity": null,
            "InitialReleaseDate": "2020-05-12T07:00:00Z",
            "CurrentReleaseDate": "2021-02-25T08:00:00Z",
            "CvrfUrl": "https://api.msrc.microsoft.com/cvrf/v2.0/document/2020-May"
        }
    ]
}

From the CVRF ID 2020-May the MSRC API provides corresponding security update CVRF JSON.

CVRF Endpoint

  
% curl -X GET --header 'Accept: application/json' 'https://api.msrc.microsoft.com/cvrf/v2.0/cvrf/2020-May'

The monthly security update will have detailed information about the products affected and a list of all the CVEs.

  
{
    "DocumentTitle": {
      "Value": "May 2020 Security Updates"
    },
    "DocumentType": {
      "Value": "Security Update"
    },
    "DocumentPublisher": {
      "ContactDetails": {
        "Value": "secure@microsoft.com"
      },
      "IssuingAuthority": {
        "Value": "The Microsoft Security Response Center (MSRC) identifies, monitors, resolves, and responds to security incidents and Microsoft software security vulnerabilities. For more information, see http://www.microsoft.com/security/msrc."
      },
      "Type": 0
    },
    "DocumentTracking": {
      "Identification": {
        "ID": {
          "Value": "2020-May"
        },

/* several lines omitted */

Within contains more detail about each CVE:

  
{
      "Title": {
        "Value": "Windows Print Spooler Elevation of Privilege Vulnerability"
      },
      "Notes": [
        {
          "Title": "Description",
          "Type": 2,
          "Ordinal": "0",
          "Value": "<p>An elevation of privilege vulnerability exists when the Windows Print Spooler service improperly allows arbitrary writing to the file system. An attacker who ....</p>\n"
        },
        {
          "Title": "Microsoft Windows",
          "Type": 7,
          "Ordinal": "20",
          "Value": "Microsoft Windows"
        },
        {
          "Title": "Issuing CNA",
          "Type": 8,
          "Ordinal": "30",
          "Value": "Microsoft"
        }
      ],
      "DiscoveryDateSpecified": false,
      "ReleaseDateSpecified": false,
      "CVE": "CVE-2020-1048",
      "ProductStatuses": [
        {
          "ProductID": [
            "11497",
            "11498",
            "11499",
            "11563",
            "11568",

Including details about the Microsoft Knowledge Base Articles, links to binary patch updates, and other useful fields are available for each CVE:

  
"Remediations": [
        {
          "Description": {
            "Value": "4556807"
          },
          "URL": "https://catalog.update.microsoft.com/v7/site/Search.aspx?q=KB4556807",
          "Supercedence": "4550922",
          "ProductID": [
            "11497",
            "11498",
            "11499",
            "11563"
          ],
          "Type": 2,
          "DateSpecified": false,
          "AffectedFiles": [],
          "RestartRequired": {
            "Value": "Yes"
          },
          "SubType": "Security Update"
        },
        {
          "Description": {
            "Value": "4551853"
          },
          "URL": "https://catalog.update.microsoft.com/v7/site/Search.aspx?q=KB4551853",
          "Supercedence": "4549949",

I was leveraging information from these two sources when attempting to explain CVEs and their relationships. I dug through the sites and JSON data to construct tables and graphs to explain CVE release timelines and frequency. While studying CVEs for popular topics (like the “Windows Print Spooler” and its explosion of CVEs over the past two years), watching presentations, reading slides and code, it was hard to keep track of it all. I found myself taking notes, creating list and tables, trying to keep track of each CVE (when they came out, the type of bug, and who found them).

Markdown and Mermaid.js

I write my notes (and most everything) in Markdown, along with what seems to be most of the rest of the world. While using Markdown, I discovered Mermaids.js for charts and graphs and haven’t looked back.

Mermaid.js

The ability dynamically to create charts and graphs from basic plaintext is powerful. Also, Mermaid.js is Javascript. All you need to include it in your website or application is this script tag:

  
<script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js"></script>

That also means that most popular blogging and code applications support it and you can create all the README graphs you want in Gitlab or Github.

For CVEs, when trying to represent a timeline of patch releases, I painstakingly created the following Mermaids.js Gantt chart.

  
```mermaid
gantt

title Windows Print Spooler (6.1.7601) Monthly Patches and CVEs
dateFormat  YYYY-MM-DD
axisFormat %Y-%m

section Monthly Cumulative Patches
KB4550964 :m1, 2020-04-14, 2020-05-11 
KB4556836 :m2, 2020-05-12, 2020-06-08
KB4561643 :m3, 2020-06-09, 2020-07-13
KB4565524 :m4, 2020-07-14, 2020-08-10
KB4571729 :m5, 2020-08-11, 2020-09-07
KB4577051 :m6, 2020-09-08, 2020-10-12
KB4580345 :m7, 2020-10-13, 2020-11-09
KB4586827 :m8, 2020-11-10, 2020-12-07
KB4592471 :m9, 2020-12-08, 2021-01-11
KB4598279 :m10, 2021-01-12, 2021-02-08
  
section Relevant CVEs

CVE-2020-1048 :a2, 2020-05-12, 2020-06-08
CVE-2020-1337 :a3, 2020-08-11, 2020-09-07
CVE-2020-17001 :a4, 2020-11-10, 2020-12-07

It was worth it because Mermiad.js transformed the plaintext into this:

gantt

title Windows Print Spooler (6.1.7601) Monthly Patches and CVEs 
dateFormat  YYYY-MM-DD
axisFormat %Y-%m

section Monthly Cumulative Patches
KB4550964 :m1, 2020-04-14, 2020-05-11 
KB4556836 :m2, 2020-05-12, 2020-06-08
KB4561643 :m3, 2020-06-09, 2020-07-13
KB4565524 :m4, 2020-07-14, 2020-08-10
KB4571729 :m5, 2020-08-11, 2020-09-07
KB4577051 :m6, 2020-09-08, 2020-10-12
KB4580345 :m7, 2020-10-13, 2020-11-09
KB4586827 :m8, 2020-11-10, 2020-12-07
KB4592471 :m9, 2020-12-08, 2021-01-11
KB4598279 :m10, 2021-01-12, 2021-02-08
  
section Relevant CVEs

CVE-2020-1048 :a2, 2020-05-12, 2020-06-08
CVE-2020-1337 :a3, 2020-08-11, 2020-09-07
CVE-2020-17001 :a4, 2020-11-10, 2020-12-07

Looks pretty good. You can copy and paste the plaintext anywhere and reuse that information, or tweak it without having to deal with generating or storing images. That being the case, after manually creating tables and charts for my CVE research about 10x, it was getting a bit tedious. Having to click through all the Mitre links and dig around in MSRC CVRF JSON became too much. The thought of having to create yet another CVE chart was the motivation I needed to do what every developer does when they discover a tedious, repeatable process, write a script. :)

The Idea

OK, the problem is that I have future reports to write that will include various CVE charts, and I want to save time and effort. I began with some common use cases.

Gantt Chart
- CVEs on a specific software component over time - Think of it like CVE version of Google Trends
- CVEs of a particular vulnerability class (or CWE) for a particular software over time - This can can hint at which types of bugs you are likely to find.
Markdown table
- Table of all the software component CVEs and their corresponding binary update download links - For Notes.
- Table of CVEs by researcher (see Acknowledgements) for a particular topic - This can help find other related research or posts by the researchers.

Although there are several more use cases I can think of, this first post will walk through the Gantt Chart and Markdown table generation.

Visual

OK, let’s generate some charts. Let me first show my thoughts with a nice Mermaid.js flowchart.


flowchart LR;

a[(Mitre CVEs)] <--> script;
c[(Microsoft CVRFs)] <--> script;
c1[(Other CVE Sources?)] <--> script;
e[CVE search term] --> script;

script --> f[amazing CVE markdown chart]

subgraph script
    d[magic logic]
end

Development

Before I started down my path of solving all my problems, I needed to find out if someone else already had. Checking to see what else is out there will help prevent you from reinventing the wheel, or at least provide inspiration and perhaps some lessons learned from other projects.

What Wheels Already Exists?

A quick Github search came up with:

Tool	Language
find_microsoft_kb_by_cve.py	Python
MSRCSecurityUpdates	Powershell

The Python script find_microsoft_kb_by_cve.py was a concise script that would take a CVE as input and dump a list of Knowledge Base IDs (like KB5010359) related to the CVE. Something like CVE -> CVRF ID -> List of KB links. This could actually serve as a solid starting point, providing the CVRF security update JSON for a CVE. And I could parse out what I needed from the corresponding CVRF JSON.

The Powershell code had some examples of using the CVRF security update with a template to generate a HTML page leveraging the provided JSON. This is similar to what I need to do to generate a Mermaid.js Markdown graph or chart. They had a HTML template that the script populated with elements from their CVRF security update JSON.

In a short amount of time, remembering that I am pretty green in Powershell scripting, I went with Python.

Requirements

OK, using find_microsoft_kb_by_cve.py as inspiration, I started on my journey. Starting with the two use cases above, I set out to build a Markdown table () and a Mermaid.js Gannt chart. Basically, the script receives a CVE keyword and spits out a Mermaid.js compatible chart (or rather its plaintext).

To support that, we need the following capabilities:

Build CVE list from Keyword - The ability to query Mitre CVE by keyword to obtain a list of matching CVEs
Map CVE to CVRF ID - Translate the CVE to their corresponding CVRF IDs
Parse Data - Parse out relevant data for the use case
Transform Into Markdown - Transform the data into valid markdown

Getting a list of CVEs - Beautiful Soup

To get a list of CVEs I began with cve.mitre.org. They provide a search page that allows for keyword search. The results for the keyword search is unfortunately just a basic HTML table (think <td>,<tr>, and <th>) and not easy to consume data like JSON. Apparently, they are working on it. To mitigate this, I used a beautiful library to scrape the data needed from the Mitre CVE search results and built out my own JSON structure from it.

I could transform the keyword search “Windows Print Spooler” into the following JSON.

  
'[
    {
        "id": "CVE-2022-23284",
        "description": "Windows Print Spooler Elevation of Privilege Vulnerability.  "
    },
    {
        "id": "CVE-2022-22718",
        "description": "Windows Print Spooler Elevation of Privilege Vulnerability. This CVE ID is unique from CVE-2022-21997, CVE-2022-21999, CVE-2022-22717.  "
    },
    {
        "id": "CVE-2022-22717",
        "description": "Windows Print Spooler Elevation of Privilege Vulnerability. This CVE ID is unique from CVE-2022-21997, CVE-2022-21999, CVE-2022-22718.  "
    },
    {
        "id": "CVE-2022-21999",
        "description": "Windows Print Spooler Elevation of Privilege Vulnerability. This CVE ID is unique from CVE-2022-21997, CVE-2022-22717, CVE-2022-22718.  "
    },
    {
        "id": "CVE-2022-21997",
        "description": "Windows Print Spooler Elevation of Privilege Vulnerability. This CVE ID is unique from CVE-2022-21999, CVE-2022-22717, CVE-2022-22718.  "
    },
    {
        "id": "CVE-2021-41333",
        "description": "Windows Print Spooler Elevation of Privilege Vulnerability  "
    },

 < several lines omitted >

From CVE to CVRF ID

The next step was to map the CVE to a CVRF ID. As explained above, MSRC provides the API.

https://api.msrc.microsoft.com/cvrf/v2.0/Updates(‘CVE-2020-1048’)

For each CVE, we could ask the MSRC API to get the CVRF-ID:

  
def get_cvrf_id_and_date_for_cve(cve):
    cvrf_id = None
    releaseDate = None

    year = int(cve.split('-')[1])
    # MSRC CVRF is not available before 2016
    if year > 2015:
        url = "{}Updates('{}')?api-version={}".format(msrc_api_url, str(cve),   str(datetime.datetime.now().year))
        headers = {}
        response = requests.get(url, headers=headers)

        if response.status_code == 200:
            data = json.loads(response.content)
            cvrf_id = data["value"][0]["ID"]
            releaseDate = data["value"][0]["InitialReleaseDate"]
            cvrf_url = data["value"][0]["CvrfUrl"]
    else:
        pass

    return cvrf_id,releaseDate

This would return the cvrf-id (something like ‘2020-Sep’) and a few more useful fields. With the cvrf-id we could then fetch the JSON from MSRC. I perform some simple caching so that we don’t hammer the MSRC API with requests.

Parsing out the data

The following table provides a high level look at data, where it’s from, and where it’s used.

API Endpoints	Format	Data	Used By
cve.mitre.org	html	`id`,`description`	get_json_cve_list_from_keyword
api.msrc.microsoft.com/cvrf/v2.0/Updates	json	`id`,`cvrf-id`,`release-date`	build_markdown_table_from_cves build_markdown_gantt_from_cves
api.msrc.microsoft.com/cvrf/v2.0/cvrf/	json	`release_date`,`kbs`,`acks`	build_markdown_table_from_cves

For the Markdown table, I select information available from the MSRC CVRF JSON.

For the Gannt Chart, you can glean all the needed the information from the updates endpoint.

Transform into Markdown

Building the Markdown CVE Table

The CVE Markdown table generation in build_markdown_table_from_cves uses the following headers.

  
def build_markdown_table_from_cves(cves,keyword):
    print("Building table...")

    table_list = []
    table_list.extend(['CVE','Description', 'Release Date', 'KBs', 'Acknowledgments'])

I found a basic Markdown generation library mdutils to help build markdown files. The data for each CVE comes from the MSRC CVRF JSON. The data is parsed and then dumped into a Markdown table constructor.

  
for cve in cves:
        print(cve['id'])
        cvrf_id,release_date = get_cvrf_id_and_date_for_cve(cve['id'])
        cvrf_json = get_knowledge_base_cvrf_json(cvrf_id)
        
        if cvrf_json:
            release_date = '[{}](https://msrc.microsoft.com/update-guide/en-US/vulnerability/{})'.format(release_date,cve['id'])
            kbs = {'[KB{}]({})-{}'.format(kb['Description']['Value'],kb['URL'],kb.get('FixedBuild')) for vuln in cvrf_json["Vulnerability"] if vuln["CVE"] == cve['id'] for kb in vuln["Remediations"] if (str(kb['Description']['Value']).isnumeric() and 'FixedBuild' in kb)  }
            kbs = sorted(['[{}]({}) - [KB{}]({})'.format(kb.get('FixedBuild'),'https://support.microsoft.com/help/{}'.format(kb['Description']['Value']),kb['Description']['Value'],kb['URL']) for vuln in cvrf_json["Vulnerability"] if vuln["CVE"] == cve['id'] for kb in vuln["Remediations"] if (str(kb['Description']['Value']).isnumeric() and 'catalog' in kb['URL'] )])
            acks = {'{}'.format(ack['Name'][0].get('Value')) for vuln in cvrf_json["Vulnerability"] if vuln["CVE"] == cve['id'] for ack in vuln["Acknowledgments"] }
        else:
            kbs = ''
            builds = ''
            acks = ''
                
        cve_id = '[{}](https://cve.mitre.org/cgi-bin/cvename.cgi?name={})'.format(cve['id'],cve['id'])

        table_list.extend([cve_id,cve['description'],release_date,'<details>'+'<br>'.join(kbs)+'</details>', '<br>'.join(acks).replace('\n',' ')])
        
    cve_table = Table().create_table(columns=column_len, rows=len(cves)+1, text=table_list, text_align='center')

Building the Gantt Mermaid.js Chart

The Gantt chart is generation is simply filling a Mermiad.js Gantt chart template with CVE data.

  
    gantt_template = '''
gantt

title {keyword}
dateFormat YYYY-MM-DD
axisFormat %Y-%m

section CVE Release Dates
{rows}
'''

The rows are built up from the api.msrc.microsoft.com/cvrf/v2.0/Updates endpoint that provides both cvrf-id and CVE release-date.

  
for num,cve in enumerate(cves):
        cvrf_id,release_date = get_cvrf_id_and_date_for_cve(cve['id'])
        
        #handle release dates
        if release_date is None:
            year = cve['id'].split('-')[1]
            release_date = '{}-01-01'.format(cve['id'].split('-')[1])
            cvrf_id = year
        else:
            from datetime import datetime
            release_date = datetime.strptime(release_date,'%Y-%m-%dT%H:%M:%SZ').strftime("%Y-%m-%d")

        row = '{} :cve{}, {}, 30d'.format(cve['id'],num,release_date)
        sections.setdefault(cvrf_id,[]).append(row)

Working POC

And finally, with the Python script a bit over explained, here is the result.

Usage

  
% python cve_markdown_charts.py -h    
usage: cve_markdown_charts.py [-h] keyword [keyword ...]

Generate CVE Markdown Charts

positional arguments:
  keyword     The CVE keyword to chart

optional arguments:
  -h, --help  show this help message and exit

For some light scripting work, the results are pretty good.

Windows Print Spooler Charts

python cve_markdown_charts.py Windows Print Spooler 

Generates the files:

Gantt

gantt

title Windows Print Spooler
dateFormat YYYY-MM-DD
axisFormat %Y-%m

section CVE Release Dates
section 2022-Mar
CVE-2022-23284 :cve0, 2022-03-08, 30d
section 2022-Feb
CVE-2022-22718 :cve1, 2022-02-08, 30d
CVE-2022-22717 :cve2, 2022-02-08, 30d
CVE-2022-21999 :cve3, 2022-02-08, 30d
CVE-2022-21997 :cve4, 2022-02-08, 30d
section 2021-Dec
CVE-2021-41333 :cve5, 2021-12-14, 30d
section 2021-Oct
CVE-2021-41332 :cve6, 2021-10-12, 30d
CVE-2021-36970 :cve10, 2021-10-12, 30d
section 2021-Sep
CVE-2021-40447 :cve7, 2021-09-14, 30d
CVE-2021-38671 :cve8, 2021-09-14, 30d
CVE-2021-38667 :cve9, 2021-09-14, 30d
section 2021-Aug
CVE-2021-36958 :cve11, 2021-08-10, 30d
CVE-2021-36947 :cve12, 2021-08-10, 30d
CVE-2021-36936 :cve13, 2021-08-10, 30d
CVE-2021-34483 :cve15, 2021-08-10, 30d
section 2021-Jul
CVE-2021-34527 :cve14, 2021-07-13, 30d
CVE-2021-34481 :cve16, 2021-07-13, 30d
section 2021-Jun
CVE-2021-1675 :cve19, 2021-06-08, 30d
section 2021-Mar
CVE-2021-26878 :cve17, 2021-03-09, 30d
CVE-2021-1640 :cve20, 2021-03-09, 30d
section 2021-Jan
CVE-2021-1695 :cve18, 2021-01-12, 30d
section 2020-Nov
CVE-2020-17042 :cve21, 2020-11-10, 30d
CVE-2020-17014 :cve22, 2020-11-10, 30d
CVE-2020-17001 :cve23, 2020-11-10, 30d
section 2020-Sep
CVE-2020-1030 :cve27, 2020-09-08, 30d
section 2020-Aug
CVE-2020-1337 :cve24, 2020-08-11, 30d
section 2020-May
CVE-2020-1070 :cve25, 2020-05-12, 30d
CVE-2020-1048 :cve26, 2020-05-12, 30d
section 2019-Mar
CVE-2019-0759 :cve28, 2019-03-12, 30d
section 2016-Jul
CVE-2016-3239 :cve29, 2016-07-12, 30d
CVE-2016-3238 :cve30, 2016-07-12, 30d
section 2013
CVE-2013-1339 :cve31, 2013-01-01, 30d
CVE-2013-0011 :cve32, 2013-01-01, 30d
section 2012
CVE-2012-1851 :cve33, 2012-01-01, 30d
section 2010
CVE-2010-2729 :cve34, 2010-01-01, 30d
section 2009
CVE-2009-0230 :cve35, 2009-01-01, 30d
CVE-2009-0229 :cve36, 2009-01-01, 30d
CVE-2009-0228 :cve37, 2009-01-01, 30d
section 2006
CVE-2006-6296 :cve38, 2006-01-01, 30d
section 2005
CVE-2005-1984 :cve39, 2005-01-01, 30d
section 2001
CVE-2001-1451 :cve40, 2001-01-01, 30d
section 1999
CVE-1999-0899 :cve41, 1999-01-01, 30d
CVE-1999-0898 :cve42, 1999-01-01, 30d

Google Chrome UAF

python cve_markdown_charts.py Google Chrome Use After Free

Generates the files:

Sometimes the CVE keyword is too broad. For the keyword “Google Chrome Use After Free” the script still works, it just takes a bit of time. It generates a chart with 600+ CVE results! Perhaps we need some filters on date range or number of results. If you want to see the full results of the output, try it yourself or view the gist here.

Another fun thing to do is use the Mermaid.Js live editor. It can help generate images if you need them or see what different themes might look like. You can click this link that somehow contains all the info (the 600 CVE references) you need.

Mermaid.JS Live Editor with 600 Google Chrome UAF refs

Done? Not Done

Like many developers (and even those that pretend to be), I began with this intention:

graph LR;

A[Problem] --> B[Automation Idea] --> C[Develop Quick Script] --> D[Problem Solved]

But ended with this situation:


graph LR;

A[Problem] --> B[Idea] --> C[Develop] --> C1[Progressively Complex Solution];
C1 --> D[New Feature Idea] --> C;
D -.-> E[Done]

The most elusive part of any software project is getting to that last step. Done. This project is no different. Like most software projects, I started with a problem and came up with a quick solution that got me most of the way there (the 80% solution). My hope of being satisfied with a quick script faded as I thought about new use cases, new features, and converting this script into something much bigger. The typical issue of a working solution snowballing into several new ideas and features once again emerged.

Some thoughts for next steps include:

Replace cve.mitre.org web scraping with pure JSON from services.nvd.nist.gov/rest/json/cves/1.0/
Support a list of CVEs as input
Add more charts
Scrap the python script entirely and build some sort of free CVE Markdown generator service

Time and motivation will determine how quickly I accomplish these tasks, but it’s always fun to have the next project in mind. I hope I encouraged you to at least check out Mermaid.js for your next blog or research report. Until next time…

If you have thoughts, ideas, or corrections let me know @clearbluejar or create an issue.

_{Cover photo by RODNAE Productions from Pexels}

projects, cve-markdown-charts

This post is licensed under CC BY 4.0 by the author.