2016-03-01 Carlos Garnacho <carlosg@gnome.org> Release 1.7.4 tracker-miner-fs: Ignore git repositories by default Any file with a .git folder will be ignored. Feel free to change this in gsettings if you found this useful. tracker-miner-fs: Ignore #*# (vim backup) files by default Update to FTS5 Our old stale copy of the FTS3/4 module is now deleted, replaced by a shinier FTS5 embedded module. If at configure time we detect that SQLite doesn't offer the FTS5 module, we will load our own, just as we used to do with FTS4. FTS5 brings a few differences in the ways it's meant to be extended, the tokenizer has been updated to cope with the differences. Also, FTS5 offers no offsets() builtin function, nor matchinfo() which we used to implement ranking. It offers though ways to implement additional functions, and builtin rank support which can be tweaked to achieve the same functional results than we did. Other than that, the ways to interact with the FTS virtual table are roughly similar to those in FTS4, insertions and deletions have been updated to do things the FTS5 way. Since it's not worth to bump the database format (data is reproducted from the journal, so we drop some embedded data such as nie:plainTextContent), the nco:hobby property has been modified to no longer be fulltext indexed, AFAIK there's no users ever setting/ accessing that, and the FTS properties change will trigger the regeneration of the FTS view and virtual tables, resulting in a seamless update to FTS5. However, we don't leave completely unscathed from the fts3_tokenizer() change. Since the older FTS3/4 tokenizer is not registered, we can't just drop the older FTS table. So it is left dangling and never accessed again, in favor of the newer fts5 table. This is obviously not a problem when creating the database from scratch. In the way, a few bugs were found. per-property weights in ranking were being given in a scrambled way (although stable across database generations). And deletion of FTS properties (or entire rows) could result in the tokens not being fully removed from the FTS table, resulting in confused searches. These are now fixed. Impact to users of tracker should be none. All the FTS Sparql-to-SQL translation has been updated to just use FTS5 syntax and tables. libtracker-data: Handle tracker:fulltextIndexed changes We no longer have a reason to deem this change incompatible. Since no actual text is stored in the FTS tables, and it merely updates its tokenization info from the original class tables, we can just drop this info and reconstruct at will. So there's no longer need to consider tracker:fulltextIndexed changes as incompatible. 2016-02-27 Cédric Valmary <cvalmary@yahoo.fr> Updated Occitan translation 2016-02-26 Marinus Schraal <fosero@gmail.com> tracker-extract: Use tracker-guarantee to ensure a title in playlists Resort to the basename in order to guarantee a title. https://bugzilla.gnome.org/show_bug.cgi?id=761466 tracker-extract: Fix year-only date extraction in gstreamer module ISO 8601 takes hours and months starting at 1. Passing 0 results in wrong parsing of the resulting date string. https://bugzilla.gnome.org/show_bug.cgi?id=761236 2016-02-25 Carlos Garnacho <carlosg@gnome.org> libtracker-miner: lower g_critical() severity And check for another condition where it might happen (if tracker_crawler_start() happens on a non-existing dir). The severity of this situation is not really critical, so it's been lowered to a g_debug(). libtracker-miner: Cancel crawling/querying on both cancellation paths When a currently inspected folder is cancelled, we could either jump to the next pending dir (if any) or the next crawling root, we must cancel these two operations prior to both. libtracker-miner: Use g_file_equal instead of direct comparison We might be passed a non-canonical directory to file_notifier_current_root_check_remove_directory(), g_file_equal will be safer. libtracker-miner: Plug fd leak when cancelling crawlers early The enumerators would just be freed but not closed, leaking fds in the underlying implementations. libtracker-direct: Handle cancellation Always go through the async path, and manage the cancellable in both the IO scheduler and the jobs. If we try to handle cancellation in the direct path, the cursor will be unref'ed (and try to grab the DB lock again) while the lock is held. There's no way cancellation can work while holding the lock, so just removed this path. 2016-02-24 Carlos Garnacho <carlosg@gnome.org> libtracker-miner: Do not break ordering with bulk operations The way bulk operations break the ordering of the sparql updates may bring certain glitches. eg. if a delete (bulk) operation happens on a folder while the sparql buffer already contains updates corresponding to files within that folder, the way we order operations results in the delete happening first, and the inserts/updates happening afterwards. This may leave the database in inconsistent states. So make bulk operations a 1 file thing, this could maybe be smarter and try to compress consecutive similar operations, but there's not much added value in it. Plus, the bulk operations we issue in the miner would never match that criteria. Because we no longer break the ordering with bulk operations, things like the error map are no longer necessary, so we can remove some icky code here. BulkOperationMerge and its functions could be further simplified, that's left for a later cleanup. libtracker-miner: Cancel ops in current buffers if a delete affects the file If a folder being deleted affects operations currently in the currently issued tasks (eg. those we emitted ::process-file on) and writeback buffers, those operations would still attempt to proceed, with different degrees of success. libtracker-miner: Stop crawling if a delete op affects the current directory Otherwise the crawler will still attempt to go through the processed folders, adding unnecesary processing and potentially leaving inconsistent state in the TrackerFileSystem if a similar file layout appeared in the future. libtracker-miner: Fix TrackerFileNotifier cancellable maintenance Resetting and reusing is not deemed safe. It is better to create new cancellables and let the old ones be last unref'ed after async callbacks finish. libtracker-miner: Delete directory altogether on interrupted crawling We can't tell the extracted data is complete or valid, so give up on that data entirely. libtracker-miner: Set uri in warning It's useful to know the file it comes from.