2016-11-21 Carlos Garnacho Release 1.11.0 libtracker-data: Wrap BIND argument with () in SQL Constructing the query as "SELECT $bind * FROM (...)" may trigger sqlite parser overflows if the BIND form turns out to be too complex, Doing the query as "SELECT ($bind) * FROM (...)" seems to be more friendly to the parser. This error was seen in gnome-music search, where the BIND() argument is something like: BIND((IF(STRSTARTS(?title_lower, "the "), SUBSTR(?title_lower, 5), IF(STRSTARTS(?title_lower, "a "), SUBSTR(?title_lower, 3), IF(STRSTARTS(?title_lower, "an "), SUBSTR(?title_lower, 4), fn:replace(fn:lower-case(?title_lower), "^[ !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]+|[ !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]+$", "") ) ) ) ) AS ?title_collation) Which, despite the fn:replace regex argument being passed as a host parameter, seemed too much to sqlite. libtracker-sparql: Distcheck fix 2016-11-21 Marek Cernocky Updated Czech translation 2016-11-20 Carlos Garnacho libtracker-sparql: Use internal headers That's not guaranteed to be there yet at compile time, so it randomly breaks build. libtracker-miner: Use TrackerNotifier on TrackerDecorator Remove the GraphUpdated handler in favor of TrackerNotifier, it's been made to be a good fit here, and the porting results in a nice code removal. Given that the RDF types list is readwrite in TrackerDecorator, but construct-only in TrackerNotifier, this means we need to cater for changes in the RDF types list, just by creating another TrackerNotifier and dropping the older one. https://bugzilla.gnome.org/show_bug.cgi?id=773028 libtracker-sparql: Add TrackerNotifier to subscribe to change notifications This object abstracts GraphUpdated, and is now the recommended method to receive notifications upon changes in Tracker data. After creation, the caller may connect to the ::events signal. This signal packs all events received in one go, so as to avoid signal emission overhead on mass changes. The TrackerNotifier behavior can be slightly modified through the flags passed during instantiation. Eg. it can be requested to query some data (urn, location) upfront, although the API is tracker:id centric otherwise. https://bugzilla.gnome.org/show_bug.cgi?id=773028 tracker: Add remote connection support to "tracker sparql" CLI tool The -r/--remote switch can be used to specify the base url to be used in queries. https://bugzilla.gnome.org/show_bug.cgi?id=773031 libtracker-remote: Add support for application/sparql-results+xml As specified in https://www.w3.org/TR/rdf-sparql-XMLres/. This cursor implementation is able to read the XML expected under that content type. https://bugzilla.gnome.org/show_bug.cgi?id=773031 Add libtracker-remote This is yet another libtracker-sparql backend to connect to remote HTTP SPARQL endpoints. Connections are made explicitly through the tracker_sparql_connection_remote_new() API call, passing a server to connect to. This commit introduces support for application/sparql-results+json as specified in https://www.w3.org/TR/sparql11-results-json/. XML format will be handled next. Just readonly queries are supported, and provided there's no authentication schemes. https://bugzilla.gnome.org/show_bug.cgi?id=773031 tracker: Fix "tracker sql -f" It was doing nothing, missing the actual execution of the query. libtracker-fts: Ensure a sqlite3_stmt is finalized on FTS5 failure paths It is an error condition if no FTS5 API is found, it's no excuse to leave the sqlite3_stmt unfinalized though. libtracker-data: Add explicit calls to grab/release an stmt An stmt can just be grabbed once at a time, and will ref both the stmt and its DB interface for as long as the stmt is grabbed. After releasing both refs will be dropped (and stmt_is_used set back to FALSE), so it will be ensured that a database connection lives as long as there are active cursors grabbing an statement. libtracker-data: Remove global interface We no longer need to restrict to a single TrackerDBInterface for the direct connection use. This also allows to maximize the throughput in clients performing multiple simultaneous async queries, as we're going from using a single TrackerDBInterface to satisfy all requests (with the locking overhead this involves) to using multiple database connections that can run different concurrent queries each. This (and the last commits) translates to faster concurrent access, which evolves close to linearly under much higher loads, instead of exponentially. It is expected though that O(n^m) happens again if the number of operations vastly exceed the number of interfaces serving requests, but that's much farther. As an example, the numbers when running multiple concurrent queries for "SELECT ?u WHERE { ?u a rdfs:Resource }" (returning 12897 elements each in this computer) progress like this on an otherwise idle session: Simul. Queries With Without 5 0,399s 0,437s 10 0,700s 0,842s 50 3,621s 4,285s 100 7,315s 8,956s 500 36,012s 1m 24,952s 1000 1m 16,355s 8m 33,880s So as expected the numbers really shine when the thread contention is most noticeable, and quite close to linearly (eg. in the "with" column, every row is roughly 10x higher than the one corresponding to simultaneous queries/10) libtracker-data: Remove TrackerDBManager locking functions The TrackerDBManager mutex is used nowhere now, it's safe to remove. libtracker-direct: Replace TrackerDBManager lock usage with private one We need to lock here not because of the TrackerDBInterface, but because of other misc things needed at the time of constructing the SQL query (like TrackerOntologies, which constructs data on demand from the gvdb, this can't work across multiple threads). But we don't need to use the TrackerDBManager lock, it can be just a private one, and we remove another usage of this global lock. libtracker-data: Protect stmt creation with the TrackerDBInterface mutex Creating the sqlite3_stmt counts as "accessing the database handle", so make it sure they're also protected by the TrackerDBInterface mutex. libtracker-data: Use internal mutex in TrackerDBInterfaceSqlite There is no reason to use a global mutex for multiple TrackerDBInterfaces, the threading restriction in sqlite is "no multiple threads using the same sqlite3* at the same time" (being cursor included "using" there). This means we must restrict simultaneous threaded access to a single TrackerDBInterface, but multiple threads are free to use a TrackerDBInterface each without further interference. So, use a per-TrackerDBInterface mutex instead of a global one, if multiple cursors are created on the same interface, they'll backreference to it, so will be blocking on the same mutex. libtracker-data: Remove threadsafe argument This was only ever used to make libtracker-direct use protected cursors, which is now set by the TRACKER_DB_MANAGER_ENABLE_MUTEXES flag, so avoid passing this all along. libtracker-data: Add ENABLE_MUTEXES TrackerDBManager initialization flag libtracker-direct: Drop mutex around TrackerDBManager initialization It doesn't need to care about this anymore, TrackerDBManager uses an internal mutex. libtracker-data: Use internal mutex for TrackerDBManager (de)init There's no reason to handle locking externally, and this is not a frequent enough operation to care about thread contention times in eg. tracker-store, so just add an internal mutex protecting initialization. libtracker-data: Pass "readonly" argument to TrackerDBInterface constructor Instead of having two almost identical call stacks to obtain rw/ro interfaces, put it all together in one with a readonly argument. libtracker-data: Rename confusing TrackerDBStatement field Now that the TrackerDBStatement is a GInitiallyUnowned, the stmt_is_sunk boolean field is somewhat confusing. It is used to keep track of the stmts being currently in use by a TrackerDBCursor, renaming it to stmt_is_used seems more obvious. libtracker-data: Make TrackerDBStatement a GInitiallyUnowned We actually deal with it as if it were (cached stmts have one ref owned by the cache, and one by the caller. non-cached has just one reference). So make TrackerDBStatement actually an unowned type, and g_object_ref_sink() in both places, if the stmt is cached the ref will be sunk by the cache, and an extra ref added for the caller, and if non-cached the only sunk reference will be the callers'. libtracker-data: Refactor LRU sqlite3_stmt management There's 3 independent operations we do on it: lookup, insert, and move to top. These have been moved to separate functions (although the code is mostly as-is). libtracker-data: Refactor sqlite3_stmt preparation Add a helper function, and use it in the two places preparing the statement and checking for errors. libtracker-data: Cleanup function initialization We create enough functions to make it worth putting those in an array, so they can be nicely categorized too. libtracker-data: Remove unused field libtracker-data: Delete collation/locale change mechanisms Those are entirely unused, we support no live locale nor collation changes. The n_active_cursors field is now unused, although it's been left as it's useful to keep track of the amount of active statements/cursors in an interface. libtracker-data: Remove sqlite3_stmt argument It's already contained in the TrackerDBStatement for all callers, and it makes no sense passing other sqlite3_stmt than the one there. libtracker-data: Indentation fix tracker-extract: Drop string constness This string is actually copied and freed, so definitely not const. tracker-extract: Remove unused variable tracker: Avoid deprecated GSettings API And use GSettingsSchema/GSettingsSchemaSource. This code is still disgusting. libtracker-sparql-backend: Make methods possibly throw GLib.Error It was missing according to the valac warnings. tracker-extract: Deep copy GstToc According to gst_discoverer_info_get_toc(), it must be deep copied if you want to use it past the discoverer info lifetime. libtracker-extract: Pass the right struct pointer Confusion between TrackerExtractInfo and TrackerDecoratorInfo, might induce crashes. libtracker-miner: Handle extra GFileMonitorEvent values We don't use G_FILE_MONITOR_WATCH_MOVES as it doesn't add much to us, so make the extra derived event types a no-op. libtracker-miner: Drop wimax handling in TrackerMinerOnline NM 1.2 deprecated this type as it doesn't support wimax anymore, and I don't even know what it is. So it seems something we can do without, falling back to the "unknown" network type is just as safe. libtracker-data: Explicitly cast string to char* This fixes a valac warning. libtracker-data: Use right constness in some internal calls The string array is fully const (i.e. we deep copy it), so make these functions have the right constness. libtracker-data: Rename internal variable We are actually requesting the context to resolve variables to the binding sql expression, so rename to a more apt need_binding_expression. libtracker-data: Handle BIND while in group graph pattern Currently SQL construction fails while using the BIND form inside a group graph pattern, eg. after OPTIONALs. When in that stage, the OPTIONALs will be already resolved to SQL in the "FROM ..." clause as nested SELECTs. As such the generated SQL is wrong, make it handle this case by wrapping the nested SELECTs under "SELECT $bind_value, * FROM (...)" so the value is made available to the wrapping SELECT. The older code still applies when building simple "{ ... . BIND (...) }" clauses. https://bugzilla.gnome.org/show_bug.cgi?id=774251 libtracker-data: Return NULL SQL expression if the binding is not yet setup Recent vala changed property setters so they compare with the previous value before emitting anything, this triggers the property getter called at a time when we just don't have a DataTable to construct the SQL expression (and it's being set anyway). So ensure the binding is in the right state before trying to construct a SQL expression for it. tracker-needle: Fix build It seems recent vala got pickier with passing unsigned to a %d parameter. It's still right nonetheless, and changing the string is a needless translation change, so just cast to int. 2016-11-15 Marinus Schraal Use nie:title instead of nmm:albumTitle nmm:albumTitle is deprecated https://bugzilla.gnome.org/show_bug.cgi?id=773697 tracker-extract: Use date as album uri identifier Use the album creation date as part of the album uri identifier if available. This should make the separation of similar named albums even better. Also port mp3 & gstreamer to use the resource helper functon for exctracting album disc data. https://bugzilla.gnome.org/show_bug.cgi?id=773697 tracker-extract: Use albumartist if available Use albumartist in libav/vorbis/flac (resource-helpers), gstreamer & mp3 extractors if available to be part of the album(-disc) uri. This makes the uri more unique and tracker better at distinguishing separate albums and it makes all the extractors behave the same. https://bugzilla.gnome.org/show_bug.cgi?id=773697 tracker-extract: Don't mix artist and albumartist The 'album artist' is not the same thing as the 'performer' or the 'artist' metadata. In some extractors these tags however got mixed to provide fallback values for certain tags. In the long run this is not a viable approach: the tracker information should be a true representation of what is in the tags. https://bugzilla.gnome.org/show_bug.cgi?id=773697 2016-11-11 Marinus Schraal tracker-extract: Fix flac albumartistsort tag parsing Albumartist tag can have a sort variant too. https://bugzilla.gnome.org/show_bug.cgi?id=772596 2016-11-08 Marinus Schraal tracker-extract: Revert libflac as default rule Keep gstreamer as the primary extractor as long as the libflac extractor is not on-par with it. 2016-11-07 Marinus Schraal tracker-extract: Pass the flac metadata Oversight from the move to TrackerResource, the flac extractor was not passing the metadata on for further processing. https://bugzilla.gnome.org/show_bug.cgi?id=774062 tracker-extract: Albumartist may be NULL in flac album_artist is a TrackerResource, but may be NULL: do not just unref it. https://bugzilla.gnome.org/show_bug.cgi?id=774062 2016-11-04 Fabio Tomat Update Friulian translation 2016-11-03 Carlos Garnacho libtracker-data: Handle overflows on libicu-based normalization We allocate by default double the string size, which is a pretty generous allotment most usually. If that is not enough for storing the normalized string, just reallocate the buffer with the given expected size and try again. Also, refactor string normalization to a separate function, so it's shared by both tracker:normalize and tracker:unaccent. https://bugzilla.gnome.org/show_bug.cgi?id=769982 libtracker-miner: Start TrackerDecorator queries on TrackerMiner::started This should not be done in the initable vfunc, as the miner should remain idle until tracker_miner_start() is called on it. This fixes a possible race condition in tracker-extract, where this query is already in flight at the time TrackerPersistence handles blacklisting of files that caused earlier crashes of tracker-extract. So the query might contain the file that's being blacklisted right away. tracker_miner_start() is called after TrackerPersistence did the blacklisting job, so the first query will get the up-to-date info. 2016-11-03 Marinus Schraal tracker-extract: Fix album disc uri in gstreamer It should build the album disc uri with the album_artist_name. https://bugzilla.gnome.org/show_bug.cgi?id=773607 2016-11-02 Daniel Mustieles Update Spanish translation 2016-10-30 Aurimas Černius Updated Lithuanian translaton 2016-10-29 Mario Blättermann Update German translation 2016-10-28 Marek Černocký Updated Czech translation 2016-10-26 Marinus Schraal tracker-extract: Only use CUE info as last resort in gstreamer CUE data was overriding file tags in the gstreamer extractor, however CUE data is notoriously unreliable. Instead, only use the CUE data if we have nothing else. https://bugzilla.gnome.org/show_bug.cgi?id=773524 2016-10-23 Anders Jonsson Update Swedish translation 2016-10-23 Gábor Kelemen Update Hungarian translation 2016-10-19 Carlos Garnacho libtracker-data: Propagate "locale mismatch" error from locale_changed() And set the differing locales in the error message. Errors here are mainly due to 1) inconsistent locales, specifically the client running with a different locale than tracker-store, and 2) faulty apps that don't call setlocale() as appropriate. We already used to warn when locales differ, but the message wasn't that useful. Including the differing locales helps narrow down the issue. This error is mainly visible in the libtracker-direct backend, tracker-store shall handle locale changes on (re)start. configure.ac: Bump version to 1.11.0 Not much worth releasing yet, but having tracker-1.10 have a higher soname than master is confusing. 2016-10-18 Piotr Drąg Update Polish translation 2016-10-18 Dz Chen Update zh_CN translation 2016-10-16 Carlos Garnacho libtracker-sparql: Avoid C++ keyword in variable "namespace" is a C++ keyword, better to avoid in public headers, spotted by Christoph Cullmann . https://bugzilla.gnome.org/show_bug.cgi?id=772979