<p dir="auto">This PR adds search functionality to the diary entries page, addressing the need for users to find entries by title and body <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="963251343" data-permission-text="Title is private" data-url="https://github.com/openstreetmap/openstreetmap-website/issues/3289" data-hovercard-type="issue" data-hovercard-url="/openstreetmap/openstreetmap-website/issues/3289/hovercard" href="https://github.com/openstreetmap/openstreetmap-website/issues/3289">#3289</a></p>
<h3 dir="auto">Key Changes</h3>
<ul dir="auto">
<li>
<p dir="auto"><strong>Model Updates</strong>:</p>
<ul dir="auto">
<li>Integrated the <code class="notranslate">pg_search</code> gem to enable full-text search on diary entries with relevance ranking.</li>
<li>Added a <code class="notranslate">searchable</code> column using a <code class="notranslate">tsvector</code> for efficient searching and created an index for performance optimization.</li>
</ul>
</li>
<li>
<p dir="auto"><strong>Controller Enhancements</strong>:</p>
<ul dir="auto">
<li>Modified the <code class="notranslate">index</code> action in <code class="notranslate">DiaryEntriesController</code> to support search queries, allowing users to filter results by keywords in titles and bodies, with optional language filtering.</li>
</ul>
</li>
<li>
<p dir="auto"><strong>Testing</strong>:</p>
<ul dir="auto">
<li>Added tests to verify the search functionality, including keyword search, language filtering, and basic edge cases.</li>
</ul>
</li>
</ul>
<h3 dir="auto">Context</h3>
<p dir="auto">Previous methods explored for implementing search functionality, such as the PostgreSQL LIKE operator and pg_trgm search, were either too fast but lacked relevance ranking or too slow to be practical, especially with large datasets like the diary entries (600,000+ records). For instance, LIKE provided speed but no relevance, while pg_trgm could take over 40 seconds to run on a dataset of this size, making it unfeasible. The pg_search gem, utilizing tsearch, offers a balance between relevance and performance, though it required additional database migrations and optimizations. Despite the need for these migrations, this approach serves as a "golden middle," providing a reasonable trade-off between speed and search result quality IMHO.</p>
<h3 dir="auto">Commit Summary</h3>
<ul dir="auto">
<li><strong>Add pg_search gem</strong>: Introduced <code class="notranslate">pg_search</code> for full-text search capabilities.</li>
<li><strong>Add searchable column to diary entries migration</strong>: Created a <code class="notranslate">tsvector</code> column for storing precomputed search data.</li>
<li><strong>Add index to searchable diary entries migration</strong>: Indexed the <code class="notranslate">searchable</code> column for performance improvements.</li>
<li><strong>Add search query logic to diary entry model and controller</strong>: Integrated search functionality into the model and controller.</li>
<li><strong>Add search functionality tests to DiaryEntriesController</strong>: Added basic tests to verify the new functionality.</li>
</ul>
<h3 dir="auto">Work in Progress</h3>
<p dir="auto">This is a draft PR. I want to confirm if I'm heading in the right direction. If the approach is approved, I'll add more tests for better coverage and handling of edge cases. Any comments and recommendations welcome.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/openstreetmap/openstreetmap-website/pull/5156'>https://github.com/openstreetmap/openstreetmap-website/pull/5156</a></p>
<h4>Commit Summary</h4>
<ul>
<li><a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/commits/c30d6fd3e0321d312cd3085058df9889dedb7f06" class="commit-link">c30d6fd</a> Add pg_search gem</li>
<li><a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/commits/e2ad361a6a0f508cbd2012cb97b40ae39e2eb974" class="commit-link">e2ad361</a> Add searchable column to diary entries migration</li>
<li><a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/commits/7382e32e71bd33cbbd4e5eb53a219c58bf689314" class="commit-link">7382e32</a> Add index to searchable diary entries migration</li>
<li><a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/commits/9cfcbfde237b6c94544a91d9d5a7bf473e66c974" class="commit-link">9cfcbfd</a> Add search query logic to diary entry model and controller</li>
<li><a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/commits/734bc63f7064729fadc1f580ed4c14ae8a70e52f" class="commit-link">734bc63</a> Add search functionality tests to DiaryEntriesController</li>
<li><a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/commits/b7736c7ef16a2cc143c6ddc368d0ab8277006e8c" class="commit-link">b7736c7</a> Merge branch 'openstreetmap:master' into diary-search</li>
</ul>
<h4 style="display: inline-block">File Changes </h4> <p style="display: inline-block">(<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files">7 files</a>)</p>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-d09ea66f8227784ff4393d88a19836f321c915ae10031d16c93d67e6283ab55f">Gemfile</a>
(3)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-89cade48462044ee1b672dc5f4c3ec250fbd29effcd8932096a23c1283c6731f">Gemfile.lock</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-4b73866ef57c70fca07c0f330d7a57aae939b11e4aae0514f303df3aa3762f89">app/controllers/diary_entries_controller.rb</a>
(47)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-d8b3d068c5bcf3464bff0f2273bab8d609b7f6b4afd6089020aff892dc4ce88f">app/models/diary_entry.rb</a>
(10)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-3a36c29b70235d527a52f2742a0a4b8ddad7299a8bc8581d40081d7f921445c7">db/migrate/20240903014508_add_searchable_column_to_diary_entries.rb</a>
(22)
</li>
<li>
<strong>A</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-4a09f706bce517d5f709cb2e86dfcce93efbd64bec73828f1462542d71f84d7d">db/migrate/20240903180252_add_index_to_searchable_diary_entries.rb</a>
(15)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156/files#diff-1dd5a8f580b9615769ddc19a64a98f455d9d20eb149c443f41d9d3c797d39b12">db/structure.sql</a>
(12)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/openstreetmap/openstreetmap-website/pull/5156.patch'>https://github.com/openstreetmap/openstreetmap-website/pull/5156.patch</a></li>
<li><a href='https://github.com/openstreetmap/openstreetmap-website/pull/5156.diff'>https://github.com/openstreetmap/openstreetmap-website/pull/5156.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/openstreetmap/openstreetmap-website/pull/5156">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AAK2OLIG54P3BPQAITJBGZ3ZUX7NRAVCNFSM6AAAAABNSW7QYKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGUYDGNJTGA4DEOI">unsubscribe</a>.<br />You are receiving this because you are subscribed to this thread.<img src="https://github.com/notifications/beacon/AAK2OLPO3ZF3RWJTOVONP23ZUX7NRA5CNFSM6AAAAABNSW7QYKWGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHJKOGZJU.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><openstreetmap/openstreetmap-website/pull/5156</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/openstreetmap/openstreetmap-website/pull/5156",
"url": "https://github.com/openstreetmap/openstreetmap-website/pull/5156",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>