From Similar to Correct: How We Built a Self-Improving RAG System for WyzorAI

A compliance assistant can produce a clear, confident, well-written answer and still be wrong in the way that matters most.

If a cannabis operator asks about a New Jersey requirement and the system retrieves a regulation from New York, the quality of the writing is irrelevant. The answer may sound credible, include legal terminology, and even provide a source. It is still the wrong source for the decision being made.

That was the challenge behind our work on WyzorAI, a compliance platform designed to help cannabis operators identify regulatory issues before an inspector does.

WyzorAI checks product labels, inventory records, advertisements, and standard operating procedures against state-specific regulations. It flags potential violations, explains what needs to be corrected, and generates supporting documentation that operators can share with inspectors, legal counsel, executives, or business partners.

Behind these workflows is a retrieval-augmented generation system, better known as RAG. Its job sounds simple: find the right regulatory information and give it to an AI model before the model answers.

In practice, “find the right information” became the hardest part.

This is the story of how we analyzed that problem, improved the search process, and turned a basic RAG implementation into a smarter, continuously evaluated system that became:

• more accurate;
• more consistent;
• much better at respecting regulatory jurisdictions.

A large language model can generate and explain text, but it does not automatically know every document inside a company’s private knowledge base. It also cannot be expected to remember every recent change to state regulations.

Retrieval-augmented generation, or RAG, solves this problem by connecting the AI model to an external source of knowledge.

The original RAG approach was introduced by researchers as a way to combine a model’s language capabilities with information retrieved from an external knowledge source. Its technical foundation is described in the original Retrieval-Augmented Generation paper.

How a RAG System Works

In simple terms, a RAG system works like this:

1. A user asks a question.
2. The system searches a knowledge base for relevant information.
3. It sends the most relevant passages to the AI model.
4. The model uses those passages to prepare an answer.

Imagine asking a specialist a difficult question. Instead of expecting them to answer entirely from memory, you first give them several pages from a reference library.

If you provide the right pages, they have a good chance of producing the right answer.

If you provide the wrong pages, even an experienced specialist may confidently explain the wrong rule.

That is why a production RAG system is not only an AI writing tool. It is also a search and knowledge-management system.

The quality of the final answer depends heavily on the quality of the information selected before the answer is generated.

WyzorAI operates in cannabis compliance, a particularly demanding RegTech domain. RegTech refers to technology that helps businesses understand, manage, and document regulatory requirements.

Cannabis regulations vary by state, business type, and license. They can define how products must be packaged, which warnings must appear on labels, how long security footage must be retained, what operators can say in advertisements, and which records a business must preserve.

The platform then creates a compliance packet containing findings, applicable regulatory clauses, corrective actions, and supporting records.

This means the system is not answering low-risk general-knowledge questions. Its results can influence inspections, product releases, operational procedures, and regulatory decisions. For this domain, “close enough” is not enough.

A retrieved document must match the subject of the question, the relevant state, the business type, and ideally the exact regulatory section.

The first version of the retrieval process used a familiar RAG pattern.

Documents were divided into smaller passages and converted into numerical representations that allowed the system to compare their meaning with the meaning of a user’s question.

When a question arrived, the system searched for passages that appeared semantically similar. In other words, it looked for information that discussed related concepts, even if it did not use exactly the same words. This worked reasonably well for straightforward questions. But it also created a dangerous failure mode: similar information was not always the correct information.

A question about an annual report could retrieve general corporate filing requirements instead of cannabis reporting rules. A question about panic buttons at a cannabis facility could match hotel security regulations. A New Jersey question could retrieve a very similar New York document. The selected text was connected to the words in the question, but not to the actual regulatory decision the user needed to make.

Based on the evaluation reports the team ran, the earlier system sometimes retrieved general state laws, hotel regulations, or corporate filing rules instead of cannabis-specific requirements. The way documents were divided into passages also contributed to the problem. The initial system used passages of approximately 4,000 characters with a small overlap between consecutive passages.

This approach created several risks:

• A passage could contain several different subjects, making it less precise.
• An important requirement could be divided between two passages.
• A passage could lose the heading or section that explained its legal context.
• The system could retrieve one part of a rule while missing the continuation.
• A broad passage could appear relevant to many questions without fully answering any of them.

The problem was not that the language model could not write a clear answer. The problem was that it was sometimes receiving the wrong evidence.

We did not want to improve the system by repeatedly changing prompts until a few answers looked better. We needed evidence.

The team created sets of golden queries: representative questions with known expected answers, expected regulatory sources, and specific facts a correct response should include.

Each question was asked three times. The results were then evaluated across several areas:

• Accuracy of the answer
• Consistency between repeated answers
• Quality of regulatory references
• Inclusion of required facts
• Selection of the correct state
• Compliance with the expected response format

A large evaluation covered 170 questions from different compliance categories. The average results looked relatively strong: accuracy reached 84.1 out of 100, while consistency reached 89.1. However, the averages hid a meaningful problem. Fifteen questions had consistency scores below 70. That meant the system could answer a question correctly in one run and provide a weaker or incorrect answer when the same question was asked again.

In a compliance product, this kind of unpredictability creates operational risk. A system that is usually correct but occasionally changes its interpretation is difficult to trust.

The team also analyzed the knowledge base itself instead of assuming every stored document was complete and properly organized.

At the time of one evaluation:

• The knowledge base contained 1,545 documents.
• Only 60% had been assigned a compliance category.
• Approximately 10.8% had no usable extracted text.

These findings helped us distinguish between different causes of failure:

• The correct document was missing from the knowledge base.
• The document existed, but its text had not been extracted correctly.
• The text existed, but it had been divided into weak or overly broad passages.
• The correct passage was found but ranked below a similar document.
• The topic was correct, but the state was wrong.

The answer was generally correct but missed an exact number or regulatory clause. Once the errors were classified, the work became much more concrete. We were no longer trying to “make the AI smarter” in a general sense. We were fixing specific parts of the process that selected and prioritized information.

1. Reorganizing How Documents Were Divided

The team processed the documents again to create more focused passages.

The goal was to give each passage enough context to explain a requirement without combining too many unrelated rules.

This is especially important for legal and regulatory documents. A section about packaging may sit next to requirements for testing, transportation, advertising, or recordkeeping.

If too many sections are stored together, the system may identify the general subject but miss the precise requirement.

After reorganizing the passages, the knowledge base was reprocessed so the updated content could be searched.

2. Adding a Second Relevance Check

The original system relied heavily on the first search result ranking. The improved version introduced a second stage.

First, the system collects a wider group of potentially relevant passages. An AI-based ranking step then reviews those candidates and decides which ones answer the complete question most directly.

The process looks like this:

Initial search → Possible sources → Additional relevance check → Final context

The first search is fast and effective at finding possible matches. The second check is more selective.

This helped the system distinguish between a document that used similar words and a document that actually answered the question.

For example, phrases such as “annual report,” “security requirement,” and “product packaging” appear in many legal documents. The additional ranking step helps determine which document belongs to the user’s specific regulatory situation.

3. Identifying the State in the User’s Question

The next improvement added explicit jurisdiction detection.

When a question contains “NJ,” “New Jersey,” “NY,” or another state reference, the system identifies that information and uses it during the search.

This may sound like a small change, but it gives the system an important business rule:

4. Including Jurisdiction in the Relevance Check

The additional ranking stage was instructed to consider both the subject and the state.

A New York packaging regulation may look extremely similar to a New Jersey packaging question. A general semantic search may rank it highly because the same regulatory language appears in both documents.

The jurisdiction-aware ranking step recognizes that the document is related to the topic but does not apply to the user’s state.

This approach is useful beyond compliance. The same principle can be applied to systems that must distinguish between:

• Different customer accounts
• Product versions
• Countries or regions
• Internal departments
• Contract types
• Access levels

Important business restrictions should be applied while information is being selected, not after the wrong information has already reached the AI model.

5. Using Document URLs as an Additional State Signal

Not every document in the knowledge base had complete metadata. However, document URLs often contained information about the state or regulatory authority.

The improved system used those URLs as an additional signal.

If the state suggested by a document URL did not match the state in the question, the document received a lower priority.

The document was not automatically removed. It simply became less likely to rank above a relevant in-state source.

Together, the improvements created several layers of protection:

State from the question → State-aware relevance check → State from the document URL → Lower priority for mismatched sources

No single layer had to be perfect. Their combined effect made it much less likely that the system would mix regulations from different states.

The model does not independently rewrite its own production code. Instead, the evaluation process makes weaknesses visible and helps the team apply targeted improvements. That is what makes it a Self-Improving RAG System.

Its intelligence is not limited to generating answers. The broader system can measure its performance, reveal recurring search problems, and support a repeatable process for becoming more accurate. A strong RAG system is not one that never makes a mistake. It is one that makes mistakes visible and gives the team a practical way to reduce them.

Testing the Improvements

The improvements were tested using the same ten difficult questions before and after the changes.

Based on the comparison evaluation conducted by the team, the results were:

• Average accuracy: increased from 28.9 to 68.7 — an improvement of 138%
• Average consistency: increased from 75.2 to 93.2 — an improvement of 24%
• Questions with excellent accuracy: increased from 0 to 4
• Questions scoring below 70 for accuracy: decreased from 10 to 5 — a reduction of 50%
• Questions with low consistency: decreased from 2 to 0, eliminating low-consistency results

Nine of the ten questions improved in accuracy.

The individual examples explain the impact better than the aggregate numbers.

For a New Jersey packaging question, the earlier system retrieved general recycling law. The improved system found cannabis-specific packaging and biodegradable-material requirements.

For a question about panic buttons, the earlier answer discussed hotel regulations. The improved answer retrieved security requirements for cannabis facilities.

For a theft-notification question, the earlier system mixed jurisdictions and produced inconsistent reporting timelines. The improved version identified the requirement for immediate law-enforcement notification and the applicable three-hour notification timeline for the New Jersey Cannabis Regulatory Commission.

The answers also became more predictable. All ten tested questions achieved at least 85% consistency, and seven reached 90% or higher.

That does not mean the work was finished.

One question about reports to the Cannabis Regulatory Commission continued to retrieve a New York financial-reporting regulation. The question did not explicitly mention New Jersey, and the system did not yet understand that “CRC” itself was a strong New Jersey signal.

That remaining failure revealed the next improvement opportunity: connecting regulatory agencies with their jurisdictions, such as:

• CRC → New Jersey
• OCM → New York

This is what a useful evaluation process should do. It should not only show that the system improved. It should also show the team what to fix next.

Key Lessons for Building Reliable RAG Systems

Do Not Evaluate RAG Only by How Good the Answer Sounds

A fluent answer can hide a search failure.

Teams should inspect which documents were selected and confirm that they belong to the correct business context.

Treat Document Information as Part of the AI System

Information such as the following can be just as important as the document text:

• State
• Document type
• Regulatory agency
• Category
• Effective date
• License type

AI search is powerful, but important business rules should still be represented explicitly.

Add a Second Ranking Stage When the Knowledge Base Is Complex

A single search step may be enough for a small and simple knowledge base.

In regulated or multi-domain environments, an additional relevance check can help separate broadly similar information from the exact information the user needs.

Measure Consistency, Not Only Average Accuracy

Repeated tests reveal whether the system is dependable.

In operational AI, a predictable answer can be just as important as the best individual answer.

Make Evaluation Part of the Product Lifecycle

Golden queries should not be used only during initial quality assurance. They should become ongoing tests for the knowledge system.

Every important failure can become a new test case. Over time, the evaluation set becomes a record of what the system must continue to understand.

The most important change in the WyzorAI project was not a new prompt or a larger AI model.

It was a shift in how we thought about the product.

The early version was an AI assistant connected to a searchable document database. The improved version became a measurable knowledge system with:

• Better-organized content
• An additional relevance check
• Explicit jurisdiction rules
• Repeatable evaluations

That shift helped the system move from finding information that looked similar to finding information that was much more likely to be correct for the operator’s real situation.

For compliance products, internal knowledge tools, legal platforms, healthcare systems, and other applications that depend on reliable evidence, this is the difference between an impressive demonstration and software people can begin to trust.

If your AI product depends on private documents, policies, regulations, or operational knowledge, the hardest problem may not be generating an answer.

It may be building a search process that consistently finds the right evidence.

Talk to Right.Link about your AI project, and let’s turn your knowledge base into a system that becomes more useful, measurable, and reliable over time.