Skip to content

An amount outlives adoption #602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WaylandYang merged 2 commits into from
Sep 10, 2026
Merged

An amount outlives adoption #602

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1909c70
An amount outlives adoption
WaylandYang Sep 10, 2026
04c143c
An attribute reads its unit the way an edge does
WaylandYang Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion crates/utopia-extract/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ pub fn build_messages(
units and all. **A stated figure left out is the loss that costs most**: the reader \
came for those numbers, and no later step can recover one that was never written \
down.\n\
8c. A listed relation followed by {{…}} can carry those **qualifiers on the edge**: when the same sentence gives both the other entity and a figure for it — an amount, a stake, a price, a share count — write the relation with its \"object\" and put the figure in \"qualifiers\" keyed exactly as listed, **as written in the text, currency and all** (\"€30 million\", \"15亿元人民币\", never a bare number): {{\"subject\":\"Vega Capital\",\"predicate\":\"invested_in\",\"object\":\"Northwind\", \"qualifiers\":{{\"amount\":\"$5 billion\"}},…}}. Never invent a key that is not listed for that relation, and never drop the figure to keep the edge — a relation without its amount is half the sentence.
8c. A listed relation followed by {{…}} can carry those **qualifiers on the edge**: when the same sentence gives both the other entity and a figure for it — an amount, a stake, a price, a share count — write the relation with its \"object\" and put the figure in \"qualifiers\" keyed exactly as listed, **as written in the text, currency and all** (\"€30 million\", \"15亿元人民币\", never a bare number): {{\"subject\":\"Vega Capital\",\"predicate\":\"invested_in\",\"object\":\"Northwind\", \"qualifiers\":{{\"amount\":\"$5 billion\"}},…}}. Never invent a key that is not listed for that relation, and never drop the figure to keep the edge — a relation without its amount is half the sentence. A relation you name after the text (rule 8) carries its figure the same way — keyed by the listed attribute that fits it, or by the plainest word for it (\"amount\", \"stake\", \"price\") when none does.
8b. A **listed** relation also takes \"value\" when what the text gives is a \
string rather than another entity — a job title, a designation, a ticker, a \
model number. Never invent an entity for a string. And when the text introduces \
Expand Down
218 changes: 185 additions & 33 deletions crates/utopia-server/src/extraction.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1492,8 +1492,10 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
continue;
};
let mut object_value = serde_json::json!({ "value": normalized });
if let Some(u) = attr.unit.as_deref().filter(|u| !u.is_empty()) {
// 单位随事实落笔：类型上的单位以后改了，旧值仍按记录时的单位读
// 单位随事实落笔：类型上的单位以后改了，旧值仍按记录时的单位读。
// 记哪个单位照 `unit_for`——从前这里无条件盖上声明的单位，实测
//「提供 500 兆瓦的风电」被模型记成金额，再盖上 ¥ 就成了 500 块钱
if let Some(u) = unit_for(&raw, datatype, None, attr.unit.as_deref()) {
object_value["unit"] = serde_json::json!(u);
}
if await_nod {
Expand Down Expand Up @@ -2304,10 +2306,16 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
同一条边再听到一次带了金额的，是同一条边补上金额。
值照 datatype 换算，单位另记一格（与 object_value 同形）；
key 不在声明里、换不动、与已记的不一致——三种都进丢弃表，不静默 */
if let (Some(pid), Some(quals)) = (predicate_id, f.qualifiers.as_ref()) {
/* 谓词未知（0010，说法在证据上）时属性照写：值落在事实上，声明等
关系被采纳时在 `adopt` 里补。不写的话，`invested_in` 在 schema.org
库里是未知说法，一句话里的 $1.5 billion 就没有地方放——召回台上
`oh-invest` 那一条正是这么丢的 */
if let Some(quals) = f.qualifiers.as_ref() {
let pid = predicate_id;
// 克隆出这一组引用：下面撞上已有属性时要往 qualifier_defs 里追加声明
let defs: Vec<&utopia_core::models::RelationType> =
qualifier_defs.get(&pid).cloned().unwrap_or_default();
let defs: Vec<&utopia_core::models::RelationType> = pid
.and_then(|p| qualifier_defs.get(&p).cloned())
.unwrap_or_default();
/* 模型常把币种单独写成一个键（`"amount": "1500000000", "currency": "CNY"`），
而不是写进数额里。那不是一个属性，是数额的单位——先把它拿出来，
数值属性解不出单位时用它，别让它作为未知 key 进丢弃表 */
Expand Down Expand Up @@ -2343,9 +2351,16 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
所以跟自动扩本体走同一个开关 */
let adopted = match declared {
Some(d) => Some(d),
/* 谓词还未知（0010，说法在证据上）：没有关系可声明，值绑到库里
已有的属性定义、先落在事实上，采纳时 `adopt` 再补声明。这一步
不动本体，所以不看自动扩本体的开关 */
None if pid.is_none() => {
attr_by_key.get(&key.trim().to_lowercase()).copied()
}
None if kb.auto_extend_ontology => {
match attr_by_key.get(&key.trim().to_lowercase()).copied() {
Some(attr) => {
let pid = pid.expect("checked above");
match utopia_store::ontology::add_relation_qualifier(
&state.pool,
doc.kb_id,
Expand Down Expand Up @@ -2377,15 +2392,50 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
}
};
let Some(def) = adopted else {
drop_signal(
state,
/* **绑不上属性定义的数也不丢。** 库里没有这个属性（schema.org 里
`amount` 是关系不是属性）、或本体冻着不让扩——从前这里进丢弃表，
数就只剩丢弃表里的一个样例。现在照 8a 的样子落成主语上的一条
字面值事实：值按原文、单位另记一格、原词 `关系.键` 进证据的
proposed_predicate，缺的定义记进 ontology_misses（0010 的样子），
采纳时人来决定它归哪儿。图里有它、有证据、能查到 */
let wording = format!("{}.{}", f.predicate.trim(), key.trim());
let unit = raw
.as_str()
.and_then(utopia_extract::parse_leading_quantity)
.and_then(|(_, u)| u)
.or_else(|| sibling_currency.map(str::to_string));
let mut literal = serde_json::json!({ "value": raw });
if let Some(u) = unit {
literal["unit"] = serde_json::Value::String(u);
}
let _ = utopia_store::ontology::record_miss(
&state.pool,
doc.kb_id,
document_id,
utopia_store::extraction_drops::reason::QUALIFIER_UNKNOWN,
&format!("{}.{}", f.predicate, key),
Some(&raw.to_string()),
"attribute_type",
&wording,
Some(&format!("{} → {raw}", f.subject.trim())),
)
.await;
let (literal_id, _) = utopia_store::graph::insert_value_fact(
&state.pool,
doc.kb_id,
subject_id,
None,
&literal,
validity,
confidence,
)
.await?;
touched_facts.push(literal_id);
utopia_store::graph::add_evidence(
&state.pool,
literal_id,
chunk.id,
f.quote.as_deref(),
Some(wording.as_str()),
)
.await?;
tracing::debug!(kb_id = %doc.kb_id, wording = %wording, "边上的属性绑不上定义，落成主语上的字面值");
continue;
};
let dt = def.datatype.as_deref().unwrap_or("text");
Expand All @@ -2402,28 +2452,7 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
continue;
};
let mut value = serde_json::json!({ "value": normalized });
/* 单位：原文里认得出的用原文的（€、¥、%），原文里**没有任何单位记号**
才落回属性声明的缺省。原文带着一个认不出的单位（"francs"）时
**不能**拿缺省顶上——实测 `EUR 30 million` 被存成了 `$`，
`15亿元人民币` 也是；币种写错比不写更糟 */
let parsed = raw
.as_str()
.and_then(utopia_extract::parse_leading_quantity);
let raw_has_unit_token = raw.as_str().is_some_and(|t| {
t.chars().any(|c| {
c.is_alphabetic()
|| matches!(c, '$' | '€' | '£' | '¥' | '₩' | '₹' | '%')
})
});
let unit = match parsed.and_then(|(_, u)| u) {
Some(u) => Some(u),
None => match sibling_currency {
Some(c) if dt == "number" => Some(c.to_string()),
_ if raw_has_unit_token => None,
_ => def.unit.clone().filter(|u| !u.is_empty()),
},
};
if let Some(u) = unit {
if let Some(u) = unit_for(raw, dt, sibling_currency, def.unit.as_deref()) {
value["unit"] = serde_json::Value::String(u);
}
let write = utopia_store::graph::upsert_fact_qualifier(
Expand Down Expand Up @@ -2938,6 +2967,129 @@ async fn chunk_lists(
)))
}

/// 一条值该记什么单位（#600「单位是读出来的，不是猜的」，实体上的属性与边上的属性同一条规矩）。
///
/// 原文里认得出的用原文的（€、¥、%、"EUR 30 million" 的 €）；模型把币种单独写成
/// 一个键时用那个；原文里写着声明的那个单位（"4300 人" 对 "人"）也算读出来的。
/// 原文带着一个认不出的单位记号（"500 兆瓦"、"francs"）时**不能**拿声明的缺省顶上——
/// 实测 `EUR 30 million` 被存成了 `$`，`500 兆瓦` 被存成了 500 块钱；单位写错比不写更糟。
/// 只有原文完全没有单位记号（一个光秃秃的数）才落回声明的缺省。
fn unit_for(
raw: &serde_json::Value,
datatype: &str,
sibling_currency: Option<&str>,
declared: Option<&str>,
) -> Option<String> {
let declared = declared.map(str::trim).filter(|u| !u.is_empty());
let text = raw.as_str();
if let Some(u) = text
.and_then(utopia_extract::parse_leading_quantity)
.and_then(|(_, u)| u)
{
return Some(u);
}
if let (Some(c), "number") = (sibling_currency, datatype) {
return Some(c.to_string());
}
if let (Some(t), Some(d)) = (text, declared) {
if t.contains(d) {
return Some(d.to_string());
}
}
let has_unit_token = text.is_some_and(|t| {
t.chars()
.any(|c| c.is_alphabetic() || matches!(c, '$' | '€' | '£' | '¥' | '₩' | '₹' | '%'))
});
if has_unit_token {
None
} else {
declared.map(str::to_string)
}
}

#[cfg(test)]
mod unit_for_tests {
use super::unit_for;
use serde_json::{json, Value};

fn s(t: &str) -> Value {
Value::String(t.to_string())
}

#[test]
fn a_unit_is_read_from_the_text_before_anything_else() {
assert_eq!(
unit_for(&s("EUR 30 million"), "number", None, Some("$")).as_deref(),
Some("€")
);
assert_eq!(
unit_for(&s("8.6亿元"), "number", None, Some("$")).as_deref(),
Some("¥")
);
assert_eq!(
unit_for(&s("12%"), "number", None, Some("¥")).as_deref(),
Some("%")
);
assert_eq!(
unit_for(&s("$5 billion"), "number", None, None).as_deref(),
Some("$")
);
}

#[test]
fn a_sibling_currency_is_the_unit_of_a_bare_number() {
assert_eq!(
unit_for(&s("1500000000"), "number", Some("CNY"), Some("$")).as_deref(),
Some("CNY")
);
// 文本型属性没有币种可言
assert_eq!(unit_for(&s("B 轮"), "text", Some("CNY"), None), None);
}

#[test]
fn the_declared_unit_written_in_the_text_counts_as_read() {
assert_eq!(
unit_for(&s("4300 人"), "number", None, Some("人")).as_deref(),
Some("人")
);
assert_eq!(
unit_for(&s("三年"), "text", None, Some("年")).as_deref(),
Some("年")
);
}

#[test]
fn an_unknown_unit_token_is_never_overwritten_by_the_default() {
// 宽松扫描把尾巴上的记号当单位读出来：记的是原文的单位，不是声明的 ¥
assert_eq!(
unit_for(&s("500 兆瓦"), "number", None, Some("¥")).as_deref(),
Some("兆瓦")
);
assert_eq!(
unit_for(&s("30 million francs"), "number", None, Some("$")).as_deref(),
Some("francs")
);
// 扫描读不出、原文却明明带着字：也不拿缺省顶上
assert_eq!(
unit_for(&s("about five hundred"), "number", None, Some("$")),
None
);
}

#[test]
fn a_bare_figure_takes_the_declared_default() {
assert_eq!(
unit_for(&s("4300"), "number", None, Some("人")).as_deref(),
Some("人")
);
assert_eq!(
unit_for(&json!(4300), "number", None, Some("人")).as_deref(),
Some("人")
);
assert_eq!(unit_for(&s("4300"), "number", None, Some("")), None);
}
}

#[cfg(test)]
mod name_tests {
use super::{clause_suspect, is_entity_name};
Expand Down
26 changes: 26 additions & 0 deletions crates/utopia-store/src/graph.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1994,6 +1994,32 @@ async fn adopt(
.bind(old_id)
.execute(&mut *tx)
.await?;
/* **边上的属性跟着搬**（0037）。谓词还没被采纳时金额就已经落在旧行上——
一句「NVIDIA invested $1.5 billion in SB Energy」在 schema.org 库里
`invested_in` 是未知说法，钱不能等到采纳那天才有地方放。旧行作废、
新行接上，属性照证据的样子整体复制；顺手在关系上补声明——
这些属性定义本来就在库里，缺的只是关系上的一条声明 */
sqlx::query(
"INSERT INTO fact_qualifiers (fact_id, qualifier_type_id, value, entity_id)
SELECT $1, qualifier_type_id, value, entity_id
FROM fact_qualifiers WHERE fact_id = $2
ON CONFLICT DO NOTHING",
)
.bind(new_id)
.bind(old_id)
.execute(&mut *tx)
.await?;
sqlx::query(
"INSERT INTO relation_type_qualifiers (relation_type_id, qualifier_type_id)
SELECT $1, q.qualifier_type_id
FROM fact_qualifiers q JOIN relation_types r ON r.id = q.qualifier_type_id
WHERE q.fact_id = $2 AND r.kind = 'attribute' AND r.id <> $1
ON CONFLICT DO NOTHING",
)
.bind(predicate_id)
.bind(new_id)
.execute(&mut *tx)
.await?;
sqlx::query("UPDATE facts SET invalidated_at = now() WHERE id = $1")
.bind(old_id)
.execute(&mut *tx)
Expand Down
Loading
Loading