Lately, I’ve been running security assessments on various LLM applications using NVIDIA’s GARAK tool. If you haven’t come across it yet, GARAK is a powerful open-source scanner that checks LLMs for all kinds of vulnerabilities, everything from prompt injection to jailbreaks and data leakage.
The tool itself is fantastic, but there was one thing driving me crazy: the reports.
The Problem with JSONL Reports
GARAK outputs all its test results as JSONL files (JSON Lines), which are basically long text files with one JSON object per line. Great for machines, terrible for humans trying to make sense of test results.
I’d end up with these massive files full of valuable security data, but:
Couldn’t easily filter by vulnerability type
Had no way to sort or prioritize issues
Couldn’t quickly see patterns or success rates
Struggled to share the results with non-technical team members
Anyone who’s tried opening a raw JSONL file and making sense of it knows the pain I’m talking about.
The Solution: JSONL to Excel Converter
After wrestling with this problem, I finally decided to build a solution. I created a simple Python script that takes GARAK’s JSONL reports and transforms them into nicely organized Excel workbooks.
The tool
Takes any JSONL file (not just GARAK reports) and converts it to Excel
Creates multiple sheets for different views of the data
Adds proper formatting, column sizing, and filters
Generates summary sheets showing test distributions and success rates
Makes it easy to identify and prioritize security issues
Here’s what the output looks like for a typical GARAK report:
Summary sheet: Shows key fields like vulnerability type, status, and probe class
All Data sheet: Contains every single field from the original report
Status Analysis: Breaks down success/failure rates across all tests
Probe Success Rates: Shows which vulnerability types were most successful
Why This Matters
If you’re doing any kind of LLM security testing, quickly making sense of your test results is key. This simple conversion tool has saved me hours and helped me focus on real vulnerabilities instead of wrangling with report formatting.
The best part is, the code is super simple; just a few lines of Python using pandas and xlsxwriter. I’ve put it up on GitHub for anyone to use.
Wrapping Up
Sometimes the simplest tools make the biggest difference. I built this converter to scratch my own itch, and it’s been surprisingly effective at saving time and effort.
If you’re doing LLM security testing with GARAK, I hope it helps make your workflow smoother too.
Also, check out my second tool: GARAK Live Log Monitor with Highlights. It’s a bash script that lets you watch GARAK logs in real-time, automatically highlights key events, and saves a colorized log for later review or sharing.
In today’s data-driven world, businesses and organizations generate vast amounts of data every day. Cybersecurity analysts, data engineers, and database administrators are increasingly turning to Large Language Models (LLMs) to help generate complex database queries. However, these LLM-generated queries often don’t align with an organization’s specific database schema, creating a major headache for data professionals.
This is where SchemaWiseAIcomes in — a middleware tool designed to bridge the gap between generic AI outputs and the specific needs of your data infrastructure; currently in proof-of-concept stage. With SchemaWiseAI, you no longer need to manually adjust LLM-generated queries. The tool automatically transforms queries to match your exact data schema, saving time, reducing errors, and making data management easier.
What is SchemaWiseAI?
SchemaWiseAIis a middleware solution that adapts LLM-generated queries to match the unique database schemas of your organization. By ingesting your custom data structures, SchemaWiseAI ensures that every query is perfectly formatted and tailored to your needs, removing the need for manual adjustments. This powerful tool makes your data queries accurate, efficient, and easy to use, so you can focus on what matters most—getting insights from your data.
Why SchemaWiseAI?
LLMs can produce useful queries, but they often come with generic field names and structures that don’t fit your system. This mismatch requires tedious manual work to adapt each query to your specific data schema, causing unnecessary delays and increasing the chances of errors.
SchemaWiseAI solves this problem by automatically mapping field names and data structures to your custom schema. It makes sure that the queries generated by LLMs are accurate, efficient, and ready for execution in your environment, without the need for manual intervention.
Key Features of SchemaWise AI
Field Name Mapping: Automatically converts generic field names from LLM-generated queries into your custom names.
Query Transformation: Transforms AI-generated queries to fit your exact data schema.
Template-Based Query Generation: Quickly generates queries using predefined templates that match your system.
Example
The current proof-of-concept (POC) version of SchemaWiseAI includes a network proxy mapping feature. Below is a snippet of this mapping, which shows how internal field names used within the organization (on the left) are automatically mapped to new field names. For example, proxy log data with specific field names like “srcip“, “dstip“, “status“, etc., is automatically transformed and mapped to standardized names such as “src“, “dst“, “http_status“, and so on.
The final outcome of this schema transformation appears as follows:
User Prompt Request: List all HTTP GET requests with status 404 from the last hour
Using template query: sourcetype=”proxy” | where mtd=”GET” AND status=404 | stats count as request_count by url, srcip | sort -request_count
Final Query: sourcetype=”proxy” | where method=”GET” AND http_status=404 | stats count as request_count by uri, src | sort -request_count
For more transformation examples, check out Github.
Why Choose Ollama for SchemaWiseAI?
At the core of the current SchemaWiseAI is Ollama (https://ollama.com/), a powerful, local AI platform that runs models directly on your machine, ensuring security, privacy, and speed. Here’s why Ollama is the ideal platform for SchemaWiseAI:
Privacy and Security: Run AI models locally, ensuring that your sensitive data remains secure.
Customizable AI: Tailor the LLM to your specific database needs with ease.
Real-Time Performance: No cloud latency, providing fast, on-demand query generation.
Cost-Effective: Avoid high cloud processing costs by running everything on your own infrastructure.
To get started with Ollama, review my last post where I shared steps on how to install and configure Ollama on Kali.
Who Can Benefit from SchemaWiseAI?
SchemaWiseAI is designed for professionals who work with data and rely on accurate, fast, and customized queries. Key users include:
Cybersecurity Analysts: Quickly generate and refine queries for security logs and threat detection.
Data Engineers: Automate the process of adapting AI queries to fit specific database structures.
Database Administrators: Ensure that all queries are properly aligned with custom schemas, reducing errors and failures.
Business Intelligence Analysts: Easily generate optimized queries for reporting, dashboards, and insights.
Current Limitations
Support for More LLMs: Expanding beyond Ollama to include platforms like OpenAI and other popular models.
Integration with More Data Schemas: Supporting a wider range of schemas, such as Palo Alto logs, DNS logs, and Windows logs.
Improved UX/UI: Enhancements to the user interface for a more intuitive experience.
Expanded Query Optimization: More features to optimize queries for different platforms and use cases.
To manage scalability limitations: take machine learning, pattern-based learning approach, or a hybrid approach.
Conclusion: Transform Your Data Queries with SchemaWiseAI
SchemaWiseAI is the perfect solution for organizations looking to streamline their query generation process, improve query accuracy, and save time. Whether you’re a cybersecurity analyst, data engineer, or business intelligence analyst, SchemaWiseAI is designed to make working with data more efficient.
By automating the transformation of LLM-generated queries into organization-specific formats, SchemaWiseAI saves you the time and effort needed for manual adjustments. And with future features like broader LLM support, expanded schema integration, and improved user experience, SchemaWiseAI is positioned to become a game-changer in the world of data querying.
Disclosure:
Please note that some of the SchemaWiseAI code and content in this post were generated with the help of AI/Large Language Models (LLMs). The generated code and content has been carefully reviewed and adapted to ensure accuracy and relevance.
On August 1, 2016, Blue Coat, Inc. (K9’s parent company) was acquired by Symantec™. As can be imagined Blue Coat and Symantec had a handful of similar products and unfortunately, it didn’t make sense to maintain two competing products. it was decided to “end-of-life” K9 Web Protection. Effective immediately, K9 Web Protection is no longer available for purchase or download. Technical Support for K9 will end on June 30, 2019.
It is unfortunate to see K9 Web Protection go. I am not aware of an alternative free software that provides the same level of protection at a premium quality. However, for those interested in alternatives to K9 Web Protection, I would recommend you can start with Quad9 and OpenDNS Home. While neither of them provides everything that K9 did, but they still protect your system against most common online threats.
“We may think one layer of security will protect us – for example, antivirus. Unfortunately for that approach, history has proven that, although single-focus solutions are useful in stopping specific attacks, the capabilities of advanced malware are so broad that such protections inevitably fail.” – Jerry Shenk, Layered Security: Why It Works.
Making use of layered security for personal use is of the utmost importance as I have covered a couple of times in the past: here, here, and here. Just as I have done in the past, I will use this post to share another tool that you can explore to support your personal layered security strategy.
My never-ending curiosity to explore and test new technologies can sometimes lead me to stumble upon genuinely impressive solutions. Fortunately for you, I believe this tool falls into that category.
K9 Web Protection is the software that I have been testing for some months now, and I must say, I’ve been truly pleased with its results. The software falls under the Web Filter category, which places a restriction on websites that you can visit. Web Filtering is used in two major cases. The first is to permit parents to control the sort of content accessible to their children, offering their kids a safe environment to learn and explore online. The second is for businesses who wish to prevent their employees from accessing websites that do not pertain to their jobs.
However, in addition to the above-mentioned, from my experience using this software on a daily basis, I have come across other benefits:
Real-time malware protection“helps identify and block illegal or undesirable content in real time, including malware-infected sites. You also benefit from the WebPulse cloud service, a growing community of more than 62 million users who provide more than six billion real-time Web content ratings per day.”
You can learn more about web filtering and intelligence here.
Automatic content ratings“New websites and web pages are created every minute, and no one person can possibly rate or categorize all of them. To ensure protection against new or previously unrated websites, Blue Coat’s patent-pending Dynamic Real-Time Rating™ (DRTR) technology automatically determines the category of an unrated web page, and allows or blocks it according to your specifications.”
Another advantage of the K9 Web Protection is that it is backed by Blue Coat (acquired by Symantec in 2016), the leader in Web Security “with an impressive portfolio of integrated technologies serving as a trusted platform to deliver Cloud Generation Security to more than 15,000 customers worldwide.”
This solution is truly an “enterprise-class security software designed for home computers.” Also, did I mention that it’s free! “As part of the Blue Coat Community Outreach Program, K9 Web Protection is free for home use. You can also purchase a license to use K9 Web Protection for business, government, non-profit, or other use.”
I will do a quick overview of the installation and usage of the software, but you can find a well-documented quick start guide and user manual here:
Installation and Usage Overview:
At the time of writing, K9 Web Protection is available on Microsoft Windows 7 and higher and Mac OS X 10.8 (Mountain Lion) and higher.
The installation process should take a couple of minutes to complete as it is self-explanatory.
Upon completion, the application’s interface will open in your browser:
To view or modify any of the configurations, you will be prompted to enter the password you created during installation.
Here are some of the options and details you can access from the Setup page:
Web Categories to Block: choosing one of the available levels allows you to block selected categories of websites.
Time Restrictions: 3 options are available to block web access depending on the time of day. Unrestricted places no restrictions on web access. NightGuard blocks all web access during contiguous blocks of time every day. Custom enables you to choose days of the week and time periods to block all web access.
Web Site Exceptions: Allows you to create lists of websites to “always block” or “always allow.” Blocking Effects: “Bark When Blocked” plays a barking sound when a web page is blocked. Make sure the sound is enabled and not muted. Show Admin Options displays options on blocked web pages which enable administrators to view the blocked web page. Enable Time Out allows you to block all web access if too many web pages are blocked in a given period of time
URL Keywords: Allows you to enter keywords which, if found in a URL, cause a “block page” to display. Safe Search: “Redirect to K9 Safe Search” will redirect searches to various search engines through K9’s Safe Search. This provides a safer search experience than other search engines provide. Force Safe Search will prevent users from disabling Safe Search functionality provided by various websites.
Other Settings: “Update to Beta” enables you to get advance copies of new K9 Web Protection software undergoing development. Blue Coat distributes Beta versions so that K9 gets used in “real world” environments before being released as a final version. Please note that Beta versions might be incomplete and less stable than final versions. “Filter Secure Traffic” enables K9 to block secure websites (i.e. sites that use the HTTPS protocol).
Password/Email: Allows you to change your K9 administrator password or e-mail address.
K9 Update: Installs software updates if available.
View Activity Summary: This tab shows a summary of all “Web Activity” on your computer: To view more details, click the “Category” or “Requests” links. On these pages, you have the option of grouping the data by month or by day. To view Administrative Events details, click the “View All” link. (Some of these activities are as a result of automatic browser and toolbar updates, for example, and might display URL formats with which you are not familiar.) By selecting “Clear Logs”, all your activity data will be cleared; however, three days’ worth of administrative events will be retained.
As you can see from the above, the information provided here is extremely granular and it allows you to not only get an easy view of your browsing behavior but also the behaviors of the various system and application components. I have been using this solution in conjunction with other traditional protective mechanisms, such as anti-virus, and the benefits have been massive.
For instance, sometimes, while surfing the internet, I would see a certain URL get blocked or a visit history to a certain category in a website without a recollection of visiting that website. However, after investigations, I found that some components of a software installed on my computer or an extension in my browser is the reason behind that activity.
“The malware ecosystem has changed drastically in the past 10 years, to the point that the old precautions are just no longer enough” – Malwarebytes LABS. I have been using K9 Web Protection on many of my personal computers because I have been impressed with it, so I thought to share it here. I believe it provides that extra layer of protection that we can all appreciate in a world where cyber threats are on the rise. In addition, I believe this solution is a wonderful option for those that are less familiar with common cyber threat vectors (i.e. parents) and can easily fall for phishing emails or click on an adware as they browse the internet.
Inspired by this blog by Isaiah Sarju and this presentation given during the 2017 Denver Startup Week, I am sharing my own version: A DIY (do it yourself) Cybersecurity Guide for Startups!
This guide includes some of my favorite resources that I believe can serve as a great starting point for founders to use and build a strong security foundation for their startups.
Please make sure you check-out Isaiah’s post and the Denver presentation above; both of these are extremely thoughtful and valuable pieces!
If you found this helpful please let me know by sending me your comment and feedback below!
I plan to keep this a live list so if you know of a resource that is not already listed but will benefit others, feel free to share and I will make sure to include it!
Also, as you may know, Phishing remains as the most common tactic used by attackers to compromise both companies and individuals. “Three out of ten people will open a phishing email while one of those will proceed to click on the link, possible infecting not only their own computer but the whole firm”. – Ref.
As part of this post, I am offering a practical, hands-on training on how you can triage and respond to Phishing attacks to protect yourself, your employees and ultimately your company.
Complete the form below and let me know if you would like to learn more!
azeemnow.com 