Developer Info

Have you seen the new Ask LibreOffice support page already? http://ask.libreoffice.org

I have changed the name of the blog, to reflect the focus on the thinking behind the marketing activity at The Document Foundation. I would like to make it clear that this is a personal blog, where I will express my personal opinions. Sometimes, these opinions are shared by the other members, but quite often [...]

Like last year, again a sizeable portion of the LibreOffice community gathered in Brussels, for the habitual FOSDEM meetup with pretty much everyone who matters in Free and Open Source. This year, the typically badly overcrowded booths in the H building’s corridors got moved out to the newly requisitioned K building, among them the LibreOffice [...]

Berlin, February 14, 2012 – The Document Foundation announces LibreOffice 3.5, the third major release of “the best free office suite ever”, which shows to end users the improvements derived from the development strategy adopted since September 2010. LibreOffice 3.5 derives from the combined effort of full time hackers – the largest group of experienced [...]

Another week has passed, and time has come to write some more updates from the Intel Linux Graphics land.

As the major news, as you all already know, Mesa 8.0 was released, bringing GL 3.0 and GLSL 1.30 support combined with multitude of fixes and enhancements all over the place. For Intel i965-based graphics cards, for instance, this means a very nice boost in performance (specially on Ivy Bridge platform), lots of stability fixes and, of course, complete GL 3.0 feature set.

Besides Mesa, Kernel project was also amazingly active for the past few weeks.

• Daniel Vetter has updated his drm-intel-next tree, with a nice set of changes. Most notable of those is the support for PPGTT – which stands for Per-process Graphics Translation Table. In other words, this means that each process is able to map his own region of the GPU memory, and should not interfere with all the other ones using the GPU at the same time. This results in better performance (due to the reduced need to re-mapping addresses, and due to the fact that PPGTT entities are GPU-cacheable) and, of course, much enhanced stability.
• This new round of patches also includes a much improved interlaced modes support. These patches represent a very nice thing in our open-source community – they started with a simple request on the mailing list, which then transformed into a bugzilla entry, and after weeks of Daniel, Paulo, and many other developers working on those fixes, resulted in a amazing series of patches which greatly enhances the support for interlaced modes in Intel GPUs. This is how Open-Source community works – and this is awesome!
• Swizzling support for Sandy and Ivy Bridge architectures, which can also improve the performance for many use cases.
• Lots of bugfixes and improved debugging support.

Besides those patches, the notable patch-sets of the past weeks were:

• Ben Widawsky’s logical context switching patches. On a high-level overview of this feature, we can describe is at a feature which adds support for having a per-context set of GPU items the processes have access to. So it adds the possibility of having a different context id for each set of operations. This way, what processes in one context_id do should not affect the processes in another set. For GL_ARB_Robustness OpenGL extension, for example, it could prevent one WebGL applications from taking down all the other users of the GPU for example; and for GL_EXT_Transform_feedback it would allow each process to have a way to store its own feedback data for different stages of the pipeline – such as vertex shaders, geometry shaders, and so on.
• I have sent some patches for RC6 feature debugging, and apparently they have solved all the remaining problems with RC6 support on Sandy Bridge. So hopefully, one year after the platform launch, its remaining ghosts are finally being put to rest.

On other projects, the most notable news I managed to catch in the past weeks were some xrandr patches from Bryce Harrington from Canonical, some DRI2 enhancement patches from Mario Kleiner and input and synaptics patches from Chase Douglas on the X.org development project.

And finally, on Wayland, lots of development activity has been going on as well – most notable patches were from Ander about drag and drop icons, and patches from Juan Zhao which added support for window maximization to the core protocol.

I thought it'll be nice to blog about this before the Documentation Hackfest was started.

I am quite frustrated with the current situation with regards the API reference browsing experience in the GObject world, some of the problems are these:

• I can't quickly get a compact overview of all the properties, methods and signals of a given class
• On top of that, I need to open each super-class or interface page separately to know which properties are inherited from those
• I can't figure out all functions/methods use a given class as an argument or return type, or which classes use it as a property.
• People can't add comments or code samples to each struct/class/method/...
• People can't easily suggest amendments to the documentation without diving into the code and generating a patch + a bug in bugzilla.

On top of that, usability wise I'm not too happy with the current web frontend and/or devhelp. The information to figure out all these things are all in GIR files, so I gave it a thought and figured out what's needed. Turns out we are not too far.

So I've been developing GDN (GNOME Development Network) a little django app hosted in github using kamstrup's giraffe.  which takes GIR files and dumps them into static HTML code. The GIR parsing stuff is quite clever and useful, but we need a dynamic app if we want to provide some of the most interesting features, so I'm ussing giraffe's AST to dump all the cross referenced information into an SQL database.

So far I've managed to dump most info, the Django data model is pretty much done, I need to fix some bugs with regards to resolving specific types while crawling the parameters and return.

You need to install my personal giraffe's branch for now until Mikkel comes back from vacation and if you want it to work as I've been adding some missing features to the AST code.

So far I only have backend code no frontend code at all. Stay tuned for more info on this, and if anyone is willing to help make sure you send me an email to aruiz at gnome dot org

About LDTP:
Linux Desktop Testing Project is aimed at producing high quality test automation framework (using GNOME / Python) and cutting-edge tools that can be used to test Linux Desktop and improve it. It uses the Accessibilitylibraries to poke through the application's user interface. We strive to help in building a quality desktop.
Changes in this release:
Added Windows client version for LDTP (Hint: Expect windows version of LDTP soon ;-) ) Updated keycodes based on latest Linux distribution
Bugs fixed:
Fixed twisted gtk2 import in gtk3 environment Ubuntu 12.04 twisted XMLRPC APIs have been changed, updated accordingly
Special thanks:Mike GorseVMware Desktop UI automationTim MiaoMichael TerryBrain NitzAndre Klapper (i18n / l10n suggestions)Gurdun (To run LDTP from Robot Framework)Mardy (To run LDTP in QT environment)
Download source
Download RPMWill schedule deb build in openSUSE build service later
Documentation references:
For detailed information on LDTP framework and latest updates visit
Check LDTP APIs
Report bugs
To subscribe to LDTP mailing lists
IRC Channel - #ldtp on irc.freenode.net

I like Flask. No, really, I do. Yesterday, during a lightning talk, I claimed that I love it, and if I don't love it, then at least I love the form and function of it. I wouldn't marry it, since I don't think being married to a microframework for Web applications would provide any tax benefits. Maybe I'm getting off-topic?

Flask is built on Werkzeug, and directly uses Werkzeug routes for view lookup and URL building. As a result, anything that can be done to Werkzeug can be done to Flask. A little-known ability of Werkzeug is the ability to add new URL converters. An example and explanation is provided in the Werkzeug documentation on converters. I decided to build some cool converters which would automate some of the work I have to do when working with certain objects.

Without further ado, I would like to present ModelConverter, a class which can convert a segment of a URL representing a text field on a model into an instance of that model, and vice versa.

from __future__ import with_statement from werkzeug.routing import BaseConverter, ValidationError from sqlalchemy.orm.exc import MultipleResultsFound, NoResultFound class ModelConverter(BaseConverter): """ Converts a URL segment to and from a SQLAlchemy model. Rather than use an initializer, this class should be subclassed and have the `model` and `field` class attributes filled in. `model` is the Flask-SQLAlchemy model to use for queries, and `field` is the field on the model to use for lookups. The field to use should be Unicode or bytes. """ def to_python(self, value): try: with self.app.test_request_context(): obj = self.model.query.filter_by(**{self.field: value}).one() except (MultipleResultsFound, NoResultFound): raise ValidationError() return obj def to_url(self, value): return getattr(value, self.field)

This particular flavor uses an inheritance-based approach in order to avoid clobbering BaseConverter's initializer, but a compositional approach works too. A make_model_converter convenience method can provide the glue needed to specialize the converter. To apply it to the Flask application, merely modify the URL map after application creation:

from converters import make_model_converter from models import Character app = Flask(__name__) app.url_map.converters["character"] = make_model_converter(app, Character, "slug")

And now you can create cool things along the lines of:

@app.route("/<character:c>") def character(c): return render_template("character.html", c=c)

There is one caveat with this technique: the model instances retrieved this way will be detached from SQLAlchemy and the current session will not know about them. If you need to look up any lazily-loaded data on the models, you will need to add them to the current session first. For example, assuming Character.friends is a lazily-loaded one-to-many mapping:

@app.route("/<character:c>/friends") def character_and_friends(c): db.session.add(c) return render_template("friends.html", c=c, friends=c.friends)

Today's snippets are all real-world snippets from DCoN, and can be seen in the converters.py and views.py source files.

One of the new configuration files systemd introduced is /etc/os-release. It replaces the multitude of per-distribution release files[1] with a single one. Yesterday we decided to drop support for systems lacking /etc/os-release in systemd since recently the majority of the big distributions adopted /etc/os-release and many small ones did, too[2]. It's our hope that by dropping support for non-compliant distributions we gently put some pressure on the remaining hold-outs to adopt this scheme as well.

I'd like to take the opportunity to explain a bit what the new file offers, why application developers should care, and why the distributions should adopt it. Of course, this file is pretty much a triviality in many ways, but I guess it's still one that deserves explanation.

So, you ask why this all?

• It relieves application developers who just want to know the distribution they are running on to check for a multitude of individual release files.
• It provides both a "pretty" name (i.e. one to show to the user), and machine parsable version/OS identifiers (i.e. for use in build systems).
• It is extensible, can easily learn new fields if needed. For example, since we want to print a welcome message in the color of your distribution at boot we make it possible to configure the ANSI color for that in the file.

FAQs

There's already the lsb_release tool for this, why don't you just use that? Well, it's a very strange interface: a shell script you have to invoke (and hence spawn asynchronously from your C code), and it's not written to be extensible. It's an optional package in many distributions, and nothing we'd be happy to invoke as part of early boot in order to show a welcome message. (In times with sub-second userspace boot times we really don't want to invoke a huge shell script for a triviality like showing the welcome message). The lsb_release tool to us appears to be an attempt of abstracting distribution checks, where standardization of distribution checks is needed. It's simply a badly designed interface. In our opinion, it has its use as an interface to determine the LSB version itself, but not for checking the distribution or version.

Why haven't you adopted one of the generic release files, such as Fedora's /etc/system-release? Well, they are much nicer than lsb_release, so much is true. However, they are not extensible and are not really parsable, if the distribution needs to be identified programmatically or a specific version needs to be verified.

Why didn't you call this file /etc/bikeshed instead? The name /etc/os-release sucks! In a way, I think you kind of answered your own question there already.

Does this mean my distribution can now drop our equivalent of /etc/fedora-release? Unlikely, too much code exists that still checks for the individual release files, and you probably shouldn't break that. This new file makes things easy for applications, not for distributions: applications can now rely on a single file only, and use it in a nice way. Distributions will have to continue to ship the old files unless they are willing to break compatibility here.

This is so useless! My application needs to be compatible with distros from 1998, so how could I ever make use of the new file? I will have to continue using the old ones! True, if you need compatibility with really old distributions you do. But for new code this might not be an issue, and in general new APIs are new APIs. So if you decide to depend on it, you add a dependency on it. However, even if you need to stay compatible it might make sense to check /etc/os-release first and just fall back to the old files if it doesn't exist. The least it does for you is that you don't need 25+ open() attempts on modern distributions, but just one.

You evil people are forcing my beloved distro $XYZ to adopt your awful systemd schemes. I hate you! You hate too much, my friend. Also, I am pretty sure it's not difficult to see the benefit of this new file independently of systemd, and it's truly useful on systems without systemd, too.

I hate what you people do, can I just ignore this? Well, you really need to work on your constant feelings of hate, my friend. But, to a certain degree yes, you can ignore this for a while longer. But already, there are a number of applications making use of this file. You lose compatibility with those. Also, you are kinda working towards the further balkanization of the Linux landscape, but maybe that's your intention?

You guys add a new file because you think there are already too many? You guys are so confused! None of the existing files is generic and extensible enough to do what we want it to do. Hence we had to introduce a new one. We acknowledge the irony, however.

The file is extensible? Awesome! I want a new field XYZ= in it! Sure, it's extensible, and we are happy if distributions extend it. Please prefix your keys with your distribution's name however. Or even better: talk to us and we might be able update the documentation and make your field standard, if you convince us that it makes sense.

Anyway, to summarize all this: if you work on an application that needs to identify the OS it is being built on or is being run on, please consider making use of this new file, we created it for you. If you work on a distribution, and your distribution doesn't support this file yet, please consider adopting this file, too.

If you are working on a small/embedded distribution, or a legacy-free distribution we encourage you to adopt only this file and not establish any other per-distro release file.

Read the documentation for /etc/os-release.

Footnotes

[1] Yes, multitude, there's at least: /etc/redhat-release, /etc/SuSE-release, /etc/debian_version, /etc/arch-release, /etc/gentoo-release, /etc/slackware-version, /etc/frugalware-release, /etc/altlinux-release, /etc/mandriva-release, /etc/meego-release, /etc/angstrom-version, /etc/mageia-release. And some distributions even have multiple, for example Fedora has already four different files.

[2] To our knowledge at least OpenSUSE, Fedora, ArchLinux, Angstrom, Frugalware have adopted this. (This list is not comprehensive, there are probably more.)

So I have been using the GNOME shell for quite some time now, and I guess like any desktop experience it has its ups and downs, but at least I think I reached a stage where my remaining quibbles with the GNOME Shell is related to how it works as opposed to irritations caused by me being used to something else. One thing I keep asking myself is if GNOME Shell has in any way made me more or less productive, my guess is that the change has been mostly productivity neutral once I got used to the new setup.

That said I have some smallish irritations with the GNOME Shell, the biggest being that the menu system feels slow, it feels distinctly slower than GNOME 2.x to click into Activities->Applications->Category to get to the application I am looking for, especially if I haven’t done so in a bit and the icons have to be pulled from disk. Also the fact that the categories is on the right side of the screen means a long mouse journey across the screen to get to the categories, and then a long mouse journey usually back towards the left side of the screen to click on the application I want. Ok, so this is not an operation I perform every 5 minutes, but still it feels a bit to laborious for what it is.

The other frustration I have is with the notification dock at the bottom of the screen, but I suspect this is mostly application issues. Like for instance gtimelog seems to have a different behaviour if you click on the text or the icon, and with the icon being so small I sometimes overshot, which causes the irritation of having to move a ‘long’ way back to due the neighbouring icon having expanded. Other small irritations includes the the Banshee icon saying ‘notify-sharp’ instead of Banshee and getting a ton of notification messages on the status bar as peoples IM client set them to offline/away etc.

Hmm, actually having written my irritations down I do feel they feel rather small and insignificant, yet if someone sees this I hope these items will be improved upon for future versions.

Also I do think that the system menu should offer shut down/restart by default, if I hadn’t seen someone mentioning it on IRC some Months back I am sure I to this day wouldn’t have realized I could press ‘alt’ to get shutdown/reboot to appear in the menu. I know there is an extension now, but it is such a basic operation that should require knowing ‘secret’ buttons getting an extension.

LibreOffice 3.5 has got extraordinary typographical capabilities with the improved Graphite port of the new versions of Linux Libertine and Linux Biolinum font families: new and improved font variants from Philipp Poll et al., like the excellent Linux Libertine Display G (true size variant of Linux Libertine for 16pt or more) and the new bold [...]

see more There I Fixed It

I try to fairly regularly build recent git checkouts of all the upstream modules from X.Org (at least all those listed in the current build.sh) on Solaris. Normally I do this in 32-bit mode on x86 machines using the Sun compilers on the latest Solaris 11 internal development build, but I also occasionally do it in 64-bit mode, or with gcc compilers, or on a SPARC machine. This helps me catch issues that would break our builds when we integrate the new releases before those releases happen. (Ideally I'd set up a Solaris client of the X.Org tinderbox, but I've not gotten around to that.)

Anyways, recently I finally decided to track down an error that only shows up in the 64-bit builds of the xscope protocol monitor/decoder for X11 on Solaris. The builds run fine up until the final link stage, which fails with:

ld: fatal: relocation error: R_AMD64_PC32: file audio.o: symbol littleEndian: value 0x8086c355 does not fit ld: fatal: relocation error: R_AMD64_PC32: file audio.o: symbol ServerHostName: value 0x8086b4fe does not fit ld: fatal: relocation error: R_AMD64_PC32: file decode11.o: symbol LBXEvent: value 0x808664c3 does not fit (and over 150 more symbols that didn't fit)

A google search turned up some forum posts, a blog post, and an article on the AMD64 ABI support in the Sun Studio compilers. And indeed, the solutions they offered did work - building with -Kpic did allow the program to link.

But is that really the best answer? xscope is a simple program, and shouldn't be overflowing the normal memory model. Once it linked, looking at the resulting binary was a bit shocking:

% /usr/gnu/bin/size xscope text data bss dec hex filename 416753 5256 2155921980 2156343989 808732b5 xscope % /usr/bin/size -f xscope 23(.interp) + 32(.SUNW_cap) + 5860(.eh_frame_hdr) + 27200(.eh_frame) + 2964(.SUNW_syminfo) + 5944(.hash) + 4224(.SUNW_ldynsym) + 17784(.dynsym) + 14703(.dynstr) + 192(.SUNW_version) + 1482(.SUNW_versym) + 3168(.SUNW_dynsymsort) + 96(.SUNW_reloc) + 1944(.rela.plt) + 1312(.plt) + 291018(.text) + 33(.init) + 33(.fini) + 280(.rodata) + 38461(.rodata1) + 1376(.got) + 784(.dynamic) + 1952(.data) + 0(.bssf) + 1144(.picdata) + 0(.tdata) + 0(.tbss) + 2155921980(.bss) = 2156343989 % pmap -x `pgrep xscope` 26151: ./xscope Address Kbytes RSS Anon Locked Mode Mapped File 0000000000400000 408 408 - - r-x-- xscope 0000000000476000 8 8 8 - rw--- xscope 0000000000478000 2105388 1064 1064 - rw--- xscope 0000000080C83000 52 52 52 - rw--- [ heap ] [....] FFFFFD7FFFDF8000 32 32 32 - rw--- [ stack ] ---------------- ---------- ---------- ---------- ---------- total Kb 2108668 3204 1300 -

Two gigabytes of .bss space allocated!?!?! That can't be right. Looking through the output of the elfdump and nm programs a single symbol stood out:

Symbol Table Section: .SUNW_ldynsym index value size type bind oth ver shndx name [...] [89] 0x00000000009ff280 0x0000000080280000 OBJT GLOB D 1 .bss FDinfo [Index] Value Size Type Bind Other Shndx Name [...] [528] | 10482304| 2150105088|OBJT |GLOB |0 |28 |FDinfo

Unfortunately, that wasn't one of the ones listed in the linker errors, since it's starting address fit inside the normal memory model, but everything that came after it was out of range.

So what is this giant static allocation for? It's defined in scope.h:

#define BUFFER_SIZE (1024 \* 32) struct fdinfo { Boolean Server; long ClientNumber; FD pair; unsigned char buffer[BUFFER_SIZE]; int bufcount; int bufstart; int buflimit; /\* limited writes \*/ int bufdelivered; /\* total bytes delivered \*/ Boolean writeblocked; }; extern struct fdinfo FDinfo[StaticMaxFD];

So it allocates a 32k buffer for up to StaticMaxFD file descriptors. How many is that? For that we need to look in xscope's fd.h:

/\* need to change the MaxFD to allow larger number of fd's \*/ #define StaticMaxFD FD_SETSIZE

and from there to the Solaris system headers, which define FD_SETSIZE in <sys/select.h>:

/\* \* Select uses bit masks of file descriptors in longs. \* These macros manipulate such bit fields. \* FD_SETSIZE may be defined by the user, but the default here \* should be >= NOFILE (param.h). \*/ #ifndef FD_SETSIZE #ifdef _LP64 #define FD_SETSIZE 65536 #else #define FD_SETSIZE 1024 #endif /\* _LP64 \*/

So this makes the buffer fields alone in FDinfo become 65536 \* 32 \* 1024 bytes, aka 2 gigabytes.

Thus in this case, while compiler flags like -Kpic allow the code to link, using -DFD_SETSIZE=256 instead, builds code that's a little bit saner, fits in the normal memory model, and is less likely to fail with out of memory errors when you need it most:

% /usr/gnu/bin/size -f xscope text data bss dec hex filename 409388 3352 8449804 8862544 873b50 xscope % pmap -x `pgrep xscope` Address Kbytes RSS Anon Locked Mode Mapped File 0000000000400000 404 404 - - r-x-- xscope 0000000000475000 4 4 4 - rw--- xscope 0000000000476000 8248 20 20 - rw--- xscope 0000000000C84000 52 52 52 - rw--- [ heap ] [...] FFFFFD7FFFDFD000 12 12 12 - rw--- [ stack ] ---------------- ---------- ---------- ---------- ---------- total Kb 11500 2136 232 -

Of course that assumes that xscope is not going to be monitoring more than about 120 clients at a time (since it opens two file descriptors for each client, one connected to the client and one to the real X server), and still wastes many page mappings if you're only monitoring one client. The real fix being worked on for the next upstream release is to make the buffer allocation be dynamic, and allocate just enough for the number of clients we actually are monitoring.

The moral of this story? Just because you can make it build doesn't mean you've fixed it well, and sometimes it's useful to understand why the linker is giving you a hard time.