Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for saving .sy files in the single-line format #8712

Closed
fenilgmehta opened this issue Jul 9, 2023 · 6 comments
Closed

Support for saving .sy files in the single-line format #8712

fenilgmehta opened this issue Jul 9, 2023 · 6 comments
Assignees
Milestone

Comments

@fenilgmehta
Copy link

fenilgmehta commented Jul 9, 2023

Possible performance improvement of more than 20% at multiple places

In what scenarios do you need this feature?

Hello team,
I would like to share a performance improvement idea.

Currently, Siyuan stores a formatted version of JSON in .sy files which results in many \t and \n characters being saved to the file. Following is a table showing number of extra characters per file (for some files in SiYuan User Guide)

File name Total characters Tabs and new line % of Tabs and new line
Please Start Here 16,068 5,466 34.0%
Privacy Policy and User Agreement 10,333 2,664 25.7%
FAQ 44,585 13,988 31.3%
Acknowledgements 44,281 15,255 34.4%
Data Security 24,648 7,292 29.5%
Performance Optimization 15,687 4,367 27.8%
Glossary 21,955 4,849 22.0%

Doing the same counting for all files under SiYuan User Guide and summing it up, we get

Total characters Tabs and new line % of Tabs and new line
10,78,664 3,56,675 33%

The "Notebooks which users will create (over years)" will generally be way larger than "SiYuan User Guide". So, the percentage of "tabs and new line" characters could go even upper.

Describe the optimal solution

Based on numbers shared above, I feel if we store raw JSON (without any formatting), then it would result in:

  1. Smaller file/note size on the storage disk
  2. Faster file/note save operation
  3. Faster loading of notes
  4. Reduced memory usage when loading the notes
  5. Faster parsing of the notes/JSON when loading
  6. Reduced network load when syncing files
  7. Faster notes syncing

Most of the above improvement are linked to each other, and the primary source of all improvement is 1 (i.e. smaller file/note size on the storage disk)
The performance improvements sum up to a noticeable amount as the note/doc size increase. For example, for basic operation like opening a note, editing it, saving and syncing it will look like:

  1. Load the note (i.e. read the .sy file from the disk)
    • 🚀 faster as smaller file/note size on the storage disk, and reduced memory usage as the \t and \n characters are not loaded
  2. Parse the JSON
    • 🚀 faster as lesser number of characters to parse
  3. Render
  4. User edits the notes
  5. Save the note
    • 🚀 faster as converting JavaScript object into JSON string will be quick and lesser amount of data is to be written on the disk
  6. Sync to cloud
    • 🚀 faster as lesser amount of data to sync (due to smaller file size), and reduced network load too

I believe the expected improvement in most of the above operations is about 22% to 35% (based on the tables shown above in the previous section)

Thank You 🙏

Describe the candidate solution

No response

Other information

No response

@88250
Copy link
Member

88250 commented Jul 9, 2023

Actually, we thought about it before, but at that time because some users need to use git management (they need diff text), such as (in Chinese):

It is also because of this requirement that we retain blank characters. If we want to remove it, maybe we can only add one setting item: whether to use the single-line format, which is enabled by default in the new version.

@88250 88250 self-assigned this Jul 9, 2023
@88250 88250 added this to the 2.9.5 milestone Jul 9, 2023
@88250 88250 changed the title Possible performance improvement of more than 20% at multiple places Support for saving .sy files in the single-line format Jul 9, 2023
@88250
Copy link
Member

88250 commented Jul 9, 2023

Settings - Doc Tree - Save with a single line

image

@88250 88250 closed this as completed Jul 9, 2023
@zhoutaosheng
Copy link

Actually, we thought about it before, but at that time because some users need to use git management (they need diff text), such as (in Chinese):

It is also because of this requirement that we retain blank characters. If we want to remove it, maybe we can only add one setting item: whether to use the single-line format, which is enabled by default in the new version.

@88250 A modest proposal, "Save with a single line" is supposed to be disabled by default, because:

  1. as you have said, lots of people use git to manage .sy files, they need diff text. "Save with a single line" would lose this function
  2. readable is much more important than the size of .sy files
  3. "Save with a single line" is a new feature, should not confuse old users

@88250
Copy link
Member

88250 commented Jul 10, 2023

Yes, the added option is turned off by default, but it is turned on by default for users of newly initialized configurations.

@Zuoqiu-Yingyi
Copy link
Contributor

Zuoqiu-Yingyi commented Jul 11, 2023

@88250
It seems that this setting is not searchable in the config panel❓
image

@88250
Copy link
Member

88250 commented Jul 11, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants