Improve data indexing performance when editing doc #10976

leeyaunlong · 2024-04-11T05:56:45Z

v3.0.9 严重性能问题

Is there an existing issue for this?

I have searched the existing issues

Can the issue be reproduced with the default theme (daylight/midnight)?

I was able to reproduce the issue with the default theme

Could the issue be due to extensions?

I've ruled out the possibility that the extension is causing the problem.

Describe the problem

使用alt+[ 触发引用时,没有内容查询出来,一直在转圈loading.

Expected result

笔记量不多, 3.08和之前版本都是好的.
同时backlink也间歇性查不出内容.

Screenshot or screen recording presentation

Version environment

- Version: 3.0.9
- Operating System: win11
- Browser (if used):

Log file

W 2024/04/11 13:02:12 theme.go:159: read installed README.md failed: open ............\conf\appearance\themes\siyuan-vscodelite-edit\README_zh_CN.md: The system cannot find the file specified.
W 2024/04/11 13:05:14 theme.go:159: read installed README.md failed: open ............\conf\appearance\themes\siyuan-vscodelite-edit\README_zh_CN.md: The system cannot find the file specified.
I 2024/04/11 13:05:20 index.go:206: rebuilt database for notebook [20240411130520-fvavnz0] in [0.01s], tree [count=0, size=0 B]
I 2024/04/11 13:05:24 index.go:278: resolved refs [5005] in [3674ms]
I 2024/04/11 13:43:31 queue.go:157: database op tx [8699ms]
W 2024/04/11 13:43:47 blocktree.go:552: save block tree [size=29 MB] to [............\temp\blocktree], elapsed [2.23s]
I 2024/04/11 13:44:00 queue.go:157: database op tx [16390ms]
W 2024/04/11 13:44:00 transaction.go:138: op tx [16382ms]
W 2024/04/11 13:44:02 backlink.go:105: load ref tree [20240411134343-6ahhhzr] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-6wiq222] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-fuhdy2m] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134341-7thaubt] failed: tree not found
W 2024/04/11 13:44:05 blocktree.go:552: save block tree [size=33 MB] to [............\temp\blocktree], elapsed [2.32s]
I 2024/04/11 13:44:33 queue.go:157: database op tx [32054ms]
W 2024/04/11 13:44:33 transaction.go:138: op tx [26901ms]
W 2024/04/11 13:44:37 blocktree.go:552: save block tree [size=28 MB] to [............\temp\blocktree], elapsed [2.14s]
I 2024/04/11 13:44:53 queue.go:157: database op tx [18614ms]
W 2024/04/11 13:44:53 transaction.go:138: op tx [18360ms]
I 2024/04/11 13:45:03 queue.go:157: database op tx [8232ms]

More information

No response

leeyaunlong · 2024-04-11T06:01:31Z

很奇怪的问题

test

* https://getquicker.net/Sharedaction?code=b186d1ef-143a-41a7-bafa-08da0a4333e6
* abcdddddddddddddddddddddddd
* https://github.com/siyuan-note/siyuan/issues?page=2&q=is%3Aissue+is%3Aopen

选中 getquicker, siyuan 都查不出来. ( siyuan这还是个文档的名字, 也查不出来)
选中 cdddd.... , github 或者其他都是秒出.

UFDXD · 2024-04-11T06:31:18Z

等一下看看，可能后重建索引后，后台活动还没结束。不过目前查询的确有问题，经过发现有后台活动的话就会阻断查询一会儿

UFDXD · 2024-04-11T06:39:28Z

现在想记录东西，但后台一直不结束，有些不正常我退版本了。

88250 · 2024-04-11T07:47:01Z

这块没有改过哦，SQLite 不支持写入时查询的，这个一直都是这样。

leeyaunlong · 2024-04-11T07:56:55Z

这块没有改过哦，SQLite 不支持写入时查询的，这个一直都是这样。

试了下, 有几个不存在的笔记. 只要要这种就会卡死.

W 2024/04/11 13:44:02 backlink.go:105: load ref tree [20240411134343-6ahhhzr] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-6wiq222] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-fuhdy2m] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134341-7thaubt] failed: tree not found

必须重建索引才能解决.

但是发现这个触发了一些其他问题

比如五彩划线这个插件的同步功能, 在上面这种有不存在的索引的情况下, 只要同步就会重复创建一条笔记.点一次多创建一次.
貌似其他几个类似用到了查询的功能也都会被不存在的笔记影响功能不能用.

发生这种情况 failed: tree not found的时候能不能给用户弹个消息, 提示一下,要手工重建笔记?

ps: 我用的window 单机版, 没有开任何同步功能.

88250 · 2024-04-11T08:43:58Z

这么说应该不是性能问题而是逻辑问题？

UFDXD · 2024-04-11T08:44:51Z

不清楚，退版本正常了。之前是真的卡死

88250 · 2024-04-11T09:03:55Z

@UFDXD 你提到的问题感觉和版本没有关系，一直以来都不支持索引时查询的。

88250 · 2024-04-11T09:04:28Z

这块没有改过哦，SQLite 不支持写入时查询的，这个一直都是这样。

试了下, 有几个不存在的笔记. 只要要这种就会卡死.
W 2024/04/11 13:44:02 backlink.go:105: load ref tree [20240411134343-6ahhhzr] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-6wiq222] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-fuhdy2m] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134341-7thaubt] failed: tree not found
必须重建索引才能解决.

但是发现这个触发了一些其他问题

比如五彩划线这个插件的同步功能, 在上面这种有不存在的索引的情况下, 只要同步就会重复创建一条笔记.点一次多创建一次. 貌似其他几个类似用到了查询的功能也都会被不存在的笔记影响功能不能用.

发生这种情况 failed: tree not found的时候能不能给用户弹个消息, 提示一下,要手工重建笔记?

ps: 我用的window 单机版, 没有开任何同步功能.

所以这个应该和性能没有关系，是逻辑问题？

UFDXD · 2024-04-11T09:10:26Z

@UFDXD 你提到的问题感觉和版本没有关系，一直以来都不支持索引时查询的。

我知道这个，不过卡的没这么严重，正常也就一两秒。这次是直接转不出来了。想到是新版本，就上来看看，然后正好看到这个问题就顺便回报情况了

UFDXD · 2024-04-11T09:52:52Z

@88250 有空又上新版本了，重建索引后使用一会儿又卡死了。贴出日志

W 2024/04/11 17:34:57 blocktree.go:552: save block tree [size=155 MB] to [H:\My-Instrument\Software\思源笔记\思源工作空间\temp\blocktree], elapsed [4.29s]
I 2024/04/11 17:34:59 index.go:278: resolved refs [682] in [2048ms]
I 2024/04/11 17:37:21 queue.go:157: database op tx [141647ms]
I 2024/04/11 17:37:31 queue.go:157: database op tx [8044ms]
W 2024/04/11 17:38:12 transaction.go:138: op tx [5998ms]
W 2024/04/11 17:38:20 file.go:362: build docs [1952] elapsed [707ms]
W 2024/04/11 17:38:36 file.go:362: build docs [1953] elapsed [675ms]
I 2024/04/11 17:40:44 file.go:1616: removed doc [20220601182656-lu9smqc/20240205231813-oi4p8hu/20240411174009-961xmkb.sy]
I 2024/04/11 17:41:01 queue.go:157: database op tx [16376ms]
W 2024/04/11 17:41:01 transaction.go:138: op tx [8716ms]
W 2024/04/11 17:42:35 blocktree.go:552: save block tree [size=1.8 MB] to [H:\My-Instrument\Software\思源笔记\思源工作空间\temp\blocktree], elapsed [2.62s]
W 2024/04/11 17:45:16 transaction.go:86: file is writing:
github.com/siyuan-note/siyuan/kernel/model.WaitForWritingFiles()
D:/88250/siyuan/kernel/model/transaction.go:86 +0x85
github.com/siyuan-note/siyuan/kernel/model.createDoc({0xc03d0a35f0, 0x16}, {0xc031b85a40, 0x31}, {0xc03d0a35a0?, 0x16?}, {0xc06cb2e000, 0x0})
D:/88250/siyuan/kernel/model/file.go:1806 +0xd9a
github.com/siyuan-note/siyuan/kernel/model.createDocsByHPath({0xc03d0a35f0, 0x16}, {0xc03d0a3590, 0x17?}, {0xc06cb2e000, 0x0}, {0xc03d0a3548, 0x16}, {0xc03d0a3560, 0x16})
D:/88250/siyuan/kernel/model/path.go:110 +0x76f
github.com/siyuan-note/siyuan/kernel/model.CreateWithMarkdown({0xc03d0a3518, 0x16}, {0xc03d0a3590, 0x17}, {0x0, 0x0}, {0xc03d0a3548, 0x16}, {0xc03d0a3560, 0x16})
D:/88250/siyuan/kernel/model/file.go:1142 +0x288
github.com/siyuan-note/siyuan/kernel/api.createDocWithMd(0xc08b084b00)
D:/88250/siyuan/kernel/api/filetree.go:681 +0x46d

😅难绷

88250 · 2024-04-11T11:28:27Z

@UFDXD 感觉有点奇怪，是不是做了什么特定的操作引起的？

UFDXD · 2024-04-11T11:49:00Z

@UFDXD 感觉有点奇怪，是不是做了什么特定的操作引起的？

@88250

当时我在通过【【创建一个文档，这个文档的位置是指定的，专门放用这种方式创建的文档，也就是引用标签打tag标签收纳。

UFDXD · 2024-04-11T11:49:56Z

我退版本后，也是正常这样操作，将新版本失败的文档通过这种办法创建出来了，所以是新版本问题，也就是这一版本的改动引起的

88250 · 2024-04-11T13:06:12Z

好的，稍后我排查一下。

leeyaunlong · 2024-04-11T15:09:41Z

这块没有改过哦，SQLite 不支持写入时查询的，这个一直都是这样。

试了下, 有几个不存在的笔记. 只要要这种就会卡死.
W 2024/04/11 13:44:02 backlink.go:105: load ref tree [20240411134343-6ahhhzr] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-6wiq222] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-fuhdy2m] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134341-7thaubt] failed: tree not found
必须重建索引才能解决.

但是发现这个触发了一些其他问题

比如五彩划线这个插件的同步功能, 在上面这种有不存在的索引的情况下, 只要同步就会重复创建一条笔记.点一次多创建一次. 貌似其他几个类似用到了查询的功能也都会被不存在的笔记影响功能不能用.
发生这种情况 failed: tree not found的时候能不能给用户弹个消息, 提示一下,要手工重建笔记?

ps: 我用的window 单机版, 没有开任何同步功能.
所以这个应该和性能没有关系，是逻辑问题？

我觉得是 siyuan新版本啥底层代码逻辑遇到异常的时候没有catch导致调用这块的其他上层的代码都被卡死了.

88250 · 2024-04-11T15:11:09Z

目前找到一个数据索引慢的问题，可能是这个阻塞了其他查询。

88250 · 2024-04-11T15:47:46Z

@leeyaunlong 前面提到的问题我这里模拟测试了一下，这个警告的情况不影响上层。

W 2024/04/11 13:44:02 backlink.go:105: load ref tree [20240411134343-6ahhhzr] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-6wiq222] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134342-fuhdy2m] failed: tree not found
W 2024/04/11 13:44:03 backlink.go:105: load ref tree [20240411134341-7thaubt] failed: tree not found

这个问题暂时先忽略，这个 issue 我们先解决索引问题，解决以后你再帮忙看看是否还会卡住，谢谢。

88250 self-assigned this Apr 11, 2024

88250 added the Enhancement label Apr 11, 2024

88250 added this to the backlog milestone Apr 11, 2024

88250 changed the title ~~v3.0.9 严重性能问题~~ Improve data indexing performance Apr 11, 2024

88250 changed the title ~~Improve data indexing performance~~ Improve data indexing performance when editing doc Apr 11, 2024

88250 modified the milestones: backlog, 3.0.10 Apr 11, 2024

88250 added a commit that referenced this issue Apr 11, 2024

🎨 Improve data indexing performance when editing doc #10976

0e8f5b2

88250 closed this as completed Apr 11, 2024

88250 added a commit that referenced this issue Apr 12, 2024

🎨 Improve data indexing performance when editing doc #10976

8cd2fd7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve data indexing performance when editing doc #10976

Improve data indexing performance when editing doc #10976

leeyaunlong commented Apr 11, 2024 •

edited by 88250

leeyaunlong commented Apr 11, 2024

UFDXD commented Apr 11, 2024

UFDXD commented Apr 11, 2024

88250 commented Apr 11, 2024

leeyaunlong commented Apr 11, 2024 •

edited

88250 commented Apr 11, 2024

UFDXD commented Apr 11, 2024 •

edited

88250 commented Apr 11, 2024

88250 commented Apr 11, 2024

但是发现这个触发了一些其他问题

ps: 我用的window 单机版, 没有开任何同步功能.

UFDXD commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited

88250 commented Apr 11, 2024

UFDXD commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited

88250 commented Apr 11, 2024

leeyaunlong commented Apr 11, 2024

但是发现这个触发了一些其他问题

ps: 我用的window 单机版, 没有开任何同步功能.

88250 commented Apr 11, 2024

88250 commented Apr 11, 2024

Improve data indexing performance when editing doc #10976

Improve data indexing performance when editing doc #10976

Comments

leeyaunlong commented Apr 11, 2024 • edited by 88250

Is there an existing issue for this?

Can the issue be reproduced with the default theme (daylight/midnight)?

Could the issue be due to extensions?

Describe the problem

Expected result

Screenshot or screen recording presentation

Version environment

Log file

More information

leeyaunlong commented Apr 11, 2024

UFDXD commented Apr 11, 2024

UFDXD commented Apr 11, 2024

88250 commented Apr 11, 2024

leeyaunlong commented Apr 11, 2024 • edited

但是发现这个触发了一些其他问题

ps: 我用的window 单机版, 没有开任何同步功能.

88250 commented Apr 11, 2024

UFDXD commented Apr 11, 2024 • edited

88250 commented Apr 11, 2024

88250 commented Apr 11, 2024

但是发现这个触发了一些其他问题

ps: 我用的window 单机版, 没有开任何同步功能.

UFDXD commented Apr 11, 2024 • edited

UFDXD commented Apr 11, 2024 • edited

88250 commented Apr 11, 2024

UFDXD commented Apr 11, 2024 • edited

UFDXD commented Apr 11, 2024 • edited

88250 commented Apr 11, 2024

leeyaunlong commented Apr 11, 2024

但是发现这个触发了一些其他问题

ps: 我用的window 单机版, 没有开任何同步功能.

88250 commented Apr 11, 2024

88250 commented Apr 11, 2024

leeyaunlong commented Apr 11, 2024 •

edited by 88250

leeyaunlong commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited

UFDXD commented Apr 11, 2024 •

edited