wikipedia-kyoto-japanese-english: increase REXML entity expansion limit during XML parsing #198
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…it during XML parsing Using `Datasets::WikipediaKyotoJapaneseEnglish#each` raised an `entity expansion has grown too large (RuntimeError)`. This error occurs because the entity expansion limit in REXML is set by ruby/rexml#187, and `Datasets::WikipediaKyotoJapaneseEnglish#each` exceeds that limit. In Red Datasets, increasing the entity expansion limit is not a problem because we want to handle large datasets. Therefore, we temporarily increase the limit. How to reproduce: ```console $ cd red-datasets && bundle $ bundle exec ruby example/wikipedia-kyoto-japanese-english.rb ... /home/otegami/.rbenv/versions/3.3.3/lib/ruby/gems/3.3.0/gems/rexml-3.3.4/lib/rexml/parsers/baseparser.rb:560:in `block in unnormalize': entity expansion has grown too large (RuntimeError) ... ```
kou
reviewed
Aug 2, 2024
Comment on lines
+122
to
+128
| def with_increased_entity_expansion_text_limit | ||
| default_limit = REXML::Security.entity_expansion_text_limit | ||
| REXML::Security.entity_expansion_text_limit = ENTITY_EXPANSION_TEXT_LIMIT | ||
| yield | ||
| ensure | ||
| REXML::Security.entity_expansion_text_limit = default_limit | ||
| end |
There was a problem hiding this comment.
Hmm. We can use this for now but it seems that REXML::Parsers::StreamParser#entity_expansion_text_limit= should exist for local limitation change. Could you propose it to ruby/rexml?
There was a problem hiding this comment.
It sounds pretty nice. Sure!
There was a problem hiding this comment.
I've proposed it at ruby/rexml#192.
|
I will fix the test in other PRs. |
|
We have the same problem in |
Sure I will fix it! |
|
Can we revert this because the latest REXML fixed this problem? |
otegami
added a commit
to otegami/red-datasets
that referenced
this pull request
Aug 24, 2024
…sion limit during XML parsing (red-data-tools#198)" This reverts commit a76b917. REXML has fixed the bug where `REXML::Security.entity_expansion_text_limit` incorrectly calculated text size in both SAX and pull parsers. Therefore, we no longer need to handle this issue within Red Datasets. ref: https://github.com/ruby/rexml/releases/tag/v3.3.5
kou
pushed a commit
that referenced
this pull request
Aug 24, 2024
…sion limit during XML parsing (#198) (#201) This reverts commit a76b917. REXML has fixed the bug where `REXML::Security.entity_expansion_text_limit` incorrectly calculated text size in both SAX and pull parsers. Therefore, we no longer need to handle this issue within Red Datasets. ref: https://github.com/ruby/rexml/releases/tag/v3.3.5
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using
Datasets::WikipediaKyotoJapaneseEnglish#eachraised anentity expansion has grown too large (RuntimeError). This error occurs because the entity expansion limit in REXML is set by ruby/rexml#187, andDatasets::WikipediaKyotoJapaneseEnglish#eachexceeds that limit.In Red Datasets, increasing the entity expansion limit is not a problem because we want to handle large datasets.
Therefore, we temporarily increase the limit.
How to reproduce