Running an AI Assistant on Company Data Without the Data Leaving Vietnam

A company that has been operating for a decade holds most of what it knows in two places: files nobody has opened in years, and the heads of six people who answer the same questions every week. Somebody proposes an assistant that reads the files and answers the questions. Within about a day, the conversation stops being about retrieval quality and lands on one line in the legal review, asking where the data goes at the moment someone asks the assistant a question.

In Vietnam since 1 January 2026, that sentence has a specific answer, and the answer is not a matter of preference. Decree 356/2025/ND-CP defines cross-border transfer of personal data to include direct transfers, storage outside the country, processing on cloud infrastructure, and onward processing of data collected in Vietnam [2][8]. A question sent to a foreign model API, carrying five paragraphs of retrieved customer records, is a transfer under that definition. It is permitted. It requires consent and a documented transfer impact assessment.

So the decision has three answers, and the ranking between them moves every month while the structure holds still. Model prices fell twice in the first two weeks of August 2026. Open weights at frontier-adjacent capability shipped under an MIT licence on 31 July. GPU rental rates are drifting down as H200 inventory lands. None of that changes which three routes exist, what each one triggers legally, or the formula that tells you which one is cheaper at your volume. This post gives the formula, works one month end to end with real prices, computes the break-even volume, and then shows how far a real Vietnamese company sits from it.

The short version of the finding, before the arithmetic: at the volume a hundred-person company generates, model cost is around one million Vietnamese dong per month, and self-hosting for cost reasons is off by a factor of 155. The reason to keep processing inside Vietnam is legal, not economic. Confusing the two produces a GPU invoice that nothing justifies.

I am an engineer, not a lawyer. Everything below cites the decree or article it rests on so you can hand it to counsel and have them check my reading rather than take it.

The law that sets the boundary

Four instruments apply, and most discussions of this topic name only the first.

Three consequences follow, and they are the ones that decide architecture.

Cloud processing counts as transfer. The most common design mistake is treating a stateless API call as different from shipping a database offshore. Decree 356 does not draw that line. If a retrieved paragraph containing a customer's name and purchase history is sent to a model endpoint in Singapore or Oregon, the transfer happened, regardless of whether anything was written to disk on the other side. The obligation attaches to the processing, not to the persistence.

Whose account it is does not matter. Standing up the model behind a cloud account in the client's own name is genuinely useful: it fixes who signs the provider agreement, who holds the keys, and who can revoke access. It does nothing about location. If the region is ap-southeast-1 in Singapore, the data left Vietnam. Location is the test, and the name on the invoice is not.

A public assistant is probably medium-risk, and that means disclosure rather than residency. The AI Law's medium tier covers systems that may influence or mislead users because those users cannot recognise that they are interacting with AI or with AI-generated content [3][10]. A customer-facing assistant on a company website fits that description. The duty that follows is telling people they are talking to a machine, which is cheap. It is a separate question from where the data is processed, and conflating the two is how a project acquires a GPU it does not need. If the company operates in one of the six sectors named in Decision 33/2026, the analysis changes materially and should start with counsel rather than with an architecture diagram [5][9].

Three routes, and where the data rests

Route C is the only one that removes the transfer question outright, and it is also the only one that hands you an operational burden that never goes away. That trade is the whole decision. The cost columns above are the reason it is usually resolved the way most people do not expect, so the arithmetic comes next.

The cost of a month of questions

Monthly model cost is a product of four quantities and nothing else:

C(Q) = Q × (T_in × P_in + T_out × P_out) / 1,000,000

C(Q)   monthly model spend, USD
Q      questions answered in the month
T_in   input tokens per question
T_out  output tokens per question
P_in   vendor price per million input tokens, USD
P_out  vendor price per million output tokens, USD

T_in decomposes further, and this is where most cost estimates go wrong. An assistant that cites its sources does not send a bare question. It sends instructions, retrieved passages, and the question:

T_in = S + R + U

S   system prompt: answer scope, citation format, refusal policy
R   retrieved context: k passages × tokens per passage
U   the user's question

Take a grounded assistant with a real refusal policy, retrieving five passages of roughly 800 tokens each, answering with a source line:

The retrieved passages are 82% of the input. Any cost estimate that reasons from question length is wrong by about a factor of six.

One month worked end to end

Claude Sonnet 5 at $2.00 input and $10.00 output per million tokens [11], with 3,000 questions in the month:

input:  3,000 × 4,860 = 14,580,000 tokens = 14.580M × $2.00  = $29.16
output: 3,000 ×   400 =  1,200,000 tokens =  1.200M × $10.00 = $12.00
                                                       total  = $41.16

At 26,000 VND to the dollar, that is 1,070,160 VND per month. Roughly one million dong, for the model, for a company of a hundred people.

The same month across ten models

Per-question cost is (4,860 × P_in + 400 × P_out) / 1,000,000, so the table scales linearly in Q. Prices verified 18 August 2026 [11][12][14].

Eighty-seven times separates the cheapest row from the most expensive, for the same workload. That spread is larger than any decision you will make about retrieval chunking.

Caching takes back a third of it

The system prompt never changes, and in a corpus with a long tail some passages are retrieved far more often than others. Anthropic bills a cache read at 10% of the base input rate. Assume 2,800 of the 4,860 input tokens are stably cacheable, so effective input per question becomes 2,800 × 0.10 + 2,060 = 2,340 tokens:

Two honest caveats. Cache writes bill above the base rate, so the saving only materialises when the prefix is hit many times before it expires. And the cache has a short time-to-live, which means a low-traffic internal assistant may pay for writes it never amortises. At 3,000 questions a month spread across working hours, expect to land between the two columns rather than on the right one.

Volumes that occur in practice

An internal assistant for a hundred-person company costs less per month than the electricity for the office it sits in. The expensive part of these projects has never been the model. It is cleaning the documents, and writing down what the assistant is forbidden to answer.

Where each route wins

Break-even volume is the point where a fixed monthly hardware bill equals a linear per-question API bill:

Q* = C_self / c_api

Q*      break-even questions per month
C_self  fixed monthly cost of the self-hosted route, USD
c_api   API cost per question = (T_in × P_in + T_out × P_out) / 1,000,000

C_self has to be the loaded figure or the comparison is dishonest. One H100 SXM at RunPod's $2.69 per hour is $1,936.80 for a 720-hour month [16]. A production deployment needs a second for redundancy, and roughly a quarter of an infrastructure engineer's time. At 40,000,000 VND monthly gross for a mid-level Vietnamese infrastructure engineer, that quarter is about $385:

C_self = 2 × $1,936.80 + 0.25 × ($40,000,000 / 26,000)
       = $3,873.60 + $384.62
       = $4,258.22 per month   (110,713,600 VND)

Then Q* against each API price:

A hundred-person company generates about 2,000 questions a month. The break-even is 310,366. The gap is a factor of 155.

The DeepSeek row deserves a second look, because it is the one that ends the argument. DeepSeek publishes V4-Flash weights under an MIT licence and also serves the same model through its own API at $0.14 input and $0.28 output [14]. Self-hosting those weights does not beat paying DeepSeek to serve them until 5.4 million questions a month. Whoever runs the model at scale amortises the hardware across every customer, and a single company cannot match that on one tenant's traffic.

Capacity confirms the same thing from the other direction. One H100 running Llama 3.3 70B at FP8 sustained 1,850 to 2,400 output tokens per second at 50 to 100 concurrent requests under vLLM [17]. At 400 output tokens per answer, the low end of that band is about 12 million questions a month if the GPU never idles. At a realistic 15% average utilisation it is 1.8 million. A single rented GPU is enough for nearly any Vietnamese enterprise, which is precisely why it spends most of its life being paid for while doing nothing.

The commitment horizon, which is the part people skip

Above break-even, self-hosting wins today. The question that follows is how long today lasts, because API prices are falling and hardware commitments are not.

Model an API price as decaying geometrically per quarter:

P(m) = P₀ × (1 − r)^(m/3)

P₀  today's price per question
r   fractional price decline per quarter
m   months from now

The API reaches the self-hosted unit cost when P(m) = C_self / Q. Solving:

m = 3 × ln( (C_self / Q) / P₀ ) / ln(1 − r)

At 500,000 questions a month the self-hosted unit cost is $0.008516 against Sonnet 5's $0.01372, a ratio of 0.6207:

At 1,000,000 questions a month the horizons stretch to 33.3, 15.7, 9.8 and 5.1 months respectively.

Read that table as a constraint on contracts, not on architecture. If prices fall 20% a quarter, a hardware arrangement at 500,000 questions a month has to pay back inside six months to have been the right call. That rules out purchased servers, three-year reserved instances, and most colocation contracts. It leaves month-to-month GPU rental, which is the arrangement whose unit economics are worst. The compensating structure is a price floor: DeepSeek's own API at $0.00079 per question is 9.2% of Sonnet 5, and no plausible decay rate drives a frontier API below a rival's already-published open-weight rate quickly.

What r is in reality remains uncertain, and it would be easy to overclaim here. Two real events sit inside the last thirty days. Anthropic cancelled a scheduled increase in Sonnet 5 pricing on 10 August, keeping $2/$10 rather than moving to $3/$15 on 1 September [11]. Google shipped Gemini 3.7 Flash on 13 August at $0.75 input, half the previous rate in the same product line, 23 days after its predecessor [12][13]. Extrapolating that second step naively implies 93.6% decline per quarter, which is not a number anyone should plan against. One price step halved. Prices do not halve every 23 days. For planning, 20% per quarter is defensible and 50% is optimistic.

Thirty days of movement in the inputs

Every parameter in the sections above has a date on it. Here is what moved between 19 July and 18 August 2026, and which numbers it touches.

The pattern is one thing, and it is worth stating plainly because it drives the whole commitment-horizon argument. Two vendors within eight days chose price over margin, one by forgoing a planned increase and the other by launching at half its own previous rate. Inference is commoditising. Build for it.

The part that has to be right regardless of route

Route choice is a legal and financial question. Whether the assistant is any good is a different question, and it is answered the same way in all three cases.

An assistant that cannot name the document it relied on cannot be checked, which means nobody can tell a correct answer from a confident one. Retrieval-augmented generation exists to make that checkable: retrieve passages, pass them to the model, require the answer to cite them [18]. The requirement is not decoration. It is the only mechanism by which a domain expert can audit the system in less time than answering the question themselves.

Two conditions on the client's side determine whether the project succeeds, and neither is technical.

Somebody has to be able to judge correctness. For any assistant on specialised knowledge, the answer looks plausible to everyone except the handful of people who know the domain. If those people are not part of the build, nobody will notice the system is wrong until a customer does. This is a staffing commitment, not a review meeting.

Somebody has to write down what the assistant may not answer. Pricing, legal interpretation, dosage, warranty scope, anything with liability attached. This list belongs in the system prompt and in the evaluation set, and it belongs there before launch. An assistant with no refusal policy will answer a question about pricing by making up a price.

No assistant answers correctly every time, and any proposal that implies otherwise is describing something that does not exist. The measurable target is a stated accuracy on a fixed set of real questions, with a refusal path for everything outside scope, and a citation on every answer so a wrong one can be traced to the passage that caused it.

Setting it up

Four pieces, in the order they should be built. The point of this arrangement is that the route becomes a configuration value rather than an architectural commitment, so that when prices move again you change a line rather than a system.

Provider abstraction, so the route is a config line

LiteLLM proxies more than a hundred model providers behind one OpenAI-compatible endpoint, tracks spend per request, and runs self-hosted at no licence cost. Its enterprise tier is priced separately if you need SSO and audit logging.

pip install 'litellm[proxy]'

# litellm.config.yaml
model_list:
  - model_name: assistant          # the only name your application knows
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: assistant-cheap
    litellm_params:
      model: gemini/gemini-3.7-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: assistant-local    # route C, no data leaves the host
    litellm_params:
      model: openai/Qwen/Qwen3.8-27B
      api_base: http://gpu-host.internal:8000/v1
      api_key: none

litellm_settings:
  success_callback: ["prometheus"]  # per-request token and cost accounting
  cache: true

litellm --config litellm.config.yaml --port 4000

Your application calls assistant. Switching route means editing one line of YAML, and a like-for-like cost comparison across routes becomes a query against your own metrics rather than an argument.

Retrieval that returns a citation, not a blob

If your documents are already in Postgres, pgvector avoids introducing a second datastore and keeps everything inside one backup and one access-control boundary. Qdrant Cloud is the managed alternative, free to 1 GB and paid from roughly $25 per month for a small cluster.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE passages (
  id          bigserial PRIMARY KEY,
  doc_id      text NOT NULL,
  doc_title   text NOT NULL,
  page        int,
  updated_at  date NOT NULL,          -- so the answer can say how old the source is
  body        text NOT NULL,
  embedding   vector(1024)
);

CREATE INDEX ON passages USING hnsw (embedding vector_cosine_ops);

Carry doc_id, doc_title, page and updated_at through to the prompt, and require the model to emit them. An answer citing "Product Handbook, p. 44, updated 2024-03-11" is auditable in thirty seconds. An answer with no citation is not auditable at all.

Cost accounting from the first request

You cannot negotiate a monthly fee for something you have never measured, and this is where most projects find out their T_in assumption was wrong. Log four fields per request: input tokens, cached input tokens, output tokens, and model name. Multiply by the rate card nightly. Anthropic returns cache statistics when you mark a prefix cacheable:

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 600,
    "system": [
      {"type": "text",
       "text": "<answer scope, citation format, refusal list>",
       "cache_control": {"type": "ephemeral"}}
    ],
    "messages": [{"role": "user", "content": "<retrieved passages>\n\n<question>"}]
  }' | jq '.usage'

The usage object reports cache_creation_input_tokens and cache_read_input_tokens separately. Watch the ratio between them for a week and you will know which column of the caching table you are in.

An evaluation set of your own real questions

Public benchmarks cannot tell you whether an assistant works on your corpus, and by mid-2026 they had a second problem: OpenAI deprecated SWE-bench Verified in February over contamination concerns, and most scores in circulation are vendor self-reported [21]. Collect 100 to 200 questions people have genuinely asked, with answers your domain expert has approved, and run them on every change. promptfoo is open source and reads a plain YAML file.

npx promptfoo@latest init

# promptfooconfig.yaml
providers:
  - id: http://localhost:4000/v1/chat/completions   # via LiteLLM, so every route is testable
    config: { model: assistant }
tests:
  - vars:
      question: "Which active ingredient is in product X, and at what concentration?"
    assert:
      - type: contains
        value: "18.5%"
      - type: contains-any                          # the citation must be present
        value: ["Product Handbook", "Technical Datasheet"]
  - vars:
      question: "What discount can you give me on a 200-unit order?"
    assert:
      - type: llm-rubric
        value: "Refuses to quote a price and refers the user to a salesperson."

The second test matters more than the first. It is the refusal policy, expressed as something a build can fail on.

Route C, if you are genuinely above break-even

vLLM serves open weights with an OpenAI-compatible API, which is why it drops in behind the same LiteLLM config.

pip install vllm

vllm serve Qwen/Qwen3.8-27B \
  --quantization fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --port 8000

That server, on one rented H100 SXM at $2.69 per hour, is the $1,936.80 monthly line in the route table. Two of them plus a quarter of an engineer is the $4,258.22 that produced a break-even of 310,366 questions a month. Before provisioning either, measure Q for a month on route A. That measurement costs about $41 and settles the question.

Three sizes of the same decision

A ten-person company with no personal data in the corpus. Route A, cheapest capable model, and stop. Internal handbooks, specifications, and process documents carrying no personal data do not engage Decree 356 at all, because there is no personal data to transfer. Expect $3 to $15 a month at Gemini 3.7 Flash pricing. Do the two things that are not optional: name the person who validates answers, and write the refusal list. Do not build a vector database in week one; a hundred documents fit inside a long context window and retrieval becomes worth adding when the corpus does not.

A hundred-person, multi-site company with customer records in scope. Route B with a Vietnamese or client-controlled region, or route A with the transfer paperwork done properly. Budget $27 to $41 a month for the model at internal volume, plus one line item nobody forecasts: cleaning the documents. Get consent language and the transfer impact assessment drafted before the pilot rather than after, because that is the step that delays launch, not the retrieval quality. Measure Q from day one. If a public-facing assistant is in scope, add the AI-disclosure notice required at the medium-risk tier [3][10]. At this size, self-hosting is 155 times premature.

An enterprise in a sector named by Decision 33/2026. Banking, healthcare, education, transport, ethnicity and religion, and judicial proceedings are enumerated across 46 high-risk systems [5][9], and the obligations at that tier are not satisfied by an architecture choice. Start with counsel, then design. Route C stops being a cost question and becomes a control question: in-country processing removes the transfer analysis entirely, which is worth paying for when the alternative is a per-use-case assessment. Even here, do not buy hardware. Rent it in-country, keep the model behind a provider abstraction, and reassess quarterly, because the commitment-horizon table says a purchase has to pay back in under a year to survive contact with falling prices. FPT AI Factory and Viettel both operate in-country GPU capacity and quote by request; get a written quote and compare it against the $4,258 loaded figure above rather than against a headline hourly rate.

A decision framework

Limits of the evidence here

The parts of this that would break first, stated so you can check them rather than trust them.

Legal reading, not legal advice. I have cited the decree and article behind each claim, and I am reading them as an engineer. The scope of "processing on cloud infrastructure" in Decree 356 is broad on its face and I have not found published enforcement guidance narrowing it. Whether a specific medium-risk classification attaches to a specific assistant is a determination for counsel, and my reading of the AI Law's medium tier as covering ordinary website assistants is inference from the statutory language, not a settled position.

Token shape is an assumption, and it dominates the arithmetic. The 4,860 input tokens come from a five-passage retrieval at 800 tokens each, which is a common configuration and not a universal one. Retrieve ten passages and every cost in this post doubles. Measure your own before quoting a monthly figure to anyone.

Prices decay and this post does not. Every rate here was verified on 18 August 2026 against vendor pricing. Gemini 3.7 Flash carries a scheduled doubling on 1 January 2027 [12]. Two rates changed inside the eight days before publication. Treat the tables as a method with worked values, not as a current rate card.

The r in the payback model is not measured. I gave four scenarios and a defensible planning figure rather than a fitted value, because two price events in one month cannot support a decay rate. The naive annualisation of the Gemini step is 93.6% per quarter, which I am explicitly declining to use.

Throughput is one third-party benchmark. The 1,850 to 2,400 output tokens per second figure comes from a single versioned comparison on Llama 3.3 70B FP8 [17], not from my own measurement, and throughput on your model at your concurrency and context length will differ. It affects the capacity claim, not the break-even calculation, which rests on rental price rather than on speed.

Benchmark scores are mostly vendor self-reported. Artificial Analysis' index [15] is independent; the SWE-bench figures circulating for every model in the table above are largely not, and the benchmark was deprecated by one of its most prominent users in February 2026 over contamination [21]. Nothing in this post's decision framework depends on a leaderboard, which is deliberate.

Not verified. In-country GPU pricing from FPT AI Factory and Viettel, which I have named without quoting rates because I could not confirm a current published figure. Whether any Vietnamese hyperscaler region satisfies a given regulator, which is fact-specific. And the actual adoption rate of AI assistants among Vietnamese enterprises, for which I found no credible number.

References

1. National Assembly of Vietnam. "Law on Personal Data Protection No. 91/2025/QH15." 2025. In force 1 January 2026; raises personal-data protection to statute level.
2. Government of Vietnam. "Decree No. 356/2025/ND-CP on Personal Data Protection." 31 December 2025. Implements Law 91/2025; defines cross-border transfer to include cloud processing.
3. National Assembly of Vietnam. "Law on Artificial Intelligence No. 134/2025/QH15." 2025. In force 1 March 2026; establishes the three-tier risk framework.
4. Government of Vietnam. "Decree No. 142/2026/ND-CP guiding the Law on Artificial Intelligence." 30 April 2026. Eight chapters, 46 articles; AI sandbox and system identification codes.
5. Prime Minister of Vietnam. "Decision No. 33/2026/QD-TTg on the list of high-risk artificial intelligence systems." 30 June 2026. Enumerates 46 systems across six sectors.
6. Government of Vietnam. "Decree No. 53/2022/ND-CP guiding the Law on Cybersecurity." 2022. In-country data storage duties for cyberspace service providers.
7. DLA Piper. "Data Protection Laws of the World: Vietnam." 2026. Consolidated summary of the post-Law-91 regime.
8. Vietnam Briefing. "Vietnam's Personal Data Protection Regulation: Decree 356." 2026. Practitioner reading of the cross-border transfer scope.
9. Vietnam Briefing. "46 High-Risk AI Systems to Face Enhanced Regulatory Oversight in Vietnam." 2026. Sector breakdown of Decision 33/2026.
10. VCI Legal. "Vietnam's New AI Decree 142: What Businesses Should Know." 2026. Structure of the AI Law's risk tiers and notification path.
11. Anthropic. "Claude API Pricing." claude.com, accessed 18 August 2026. Opus 5, Sonnet 5, Fable 5, Haiku 4.5 rate card; permanence of the Sonnet 5 rate announced 10 August 2026.
12. Google. "Gemini API Pricing and Changelog." accessed 18 August 2026. Gemini 3.7 Flash introductory and post-introductory rates.
13. VentureBeat. "Google ships Gemini 3.7 Flash." 13 August 2026. Release date and positioning against the previous Flash generation.
14. DeepSeek. "DeepSeek-V4-Flash-0731 model card and API pricing." Hugging Face and deepseek.com, 31 July 2026. MIT licence, 284B mixture-of-experts with 13B active, 1M context, $0.14/$0.28.
15. Artificial Analysis. "Intelligence Index." 2026. Independent scoring placing V4-Flash third among 101 open-weight models in class.
16. ComputePrices. "RunPod vs Lambda GPU pricing comparison." updated 15 August 2026. H100 and H200 on-demand hourly rates.
17. Mark AI Code. "CUDA H100 benchmark: vLLM v0.18.0 versus TensorRT-LLM v1.2.0 on Llama 3.3 70B FP8." 2026. Output throughput at 50 to 100 concurrent requests, and engine build times.
18. Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020. arXiv:2005.11401. The grounding pattern every route in this post depends on.
19. Yahoo Finance. "OpenAI cuts GPT-5.6 Luna pricing." August 2026. Reported 80% reduction; single secondary source.
20. Fortune, citing PitchBook. "87.5% of US venture dollars went to AI." 12 August 2026. Capital concentration behind continued price competition.
21. Morph. "SWE-bench Pro and the contamination problem." 2026. Records OpenAI's February 2026 deprecation of SWE-bench Verified.

Sources: Vietnamese legal texts via thuvienphapluat.vn and the practitioner summaries cited above; model pricing verified against vendor pricing pages on 18 August 2026; GPU rental rates from ComputePrices, updated 10 to 15 August 2026; all cost, break-even and payback figures computed and cross-checked in a verification script before publication.

Last updated: August 18, 2026