Many to many search with doctrine2 and symfony2

My last post was about how to search for a many to one entity where I needed to check the values of two attributes by using "exists". I couldn't find any other "clean" ORM/DQL query to do this.

While experimenting with many to many relations I encountered how to easily retrieve all users that are associated with a certain work category. Many to many relations in doctrine create a relational table between two entities, which is no entity itself. You cannot use the table to create joins. I'd like to cite the doctrine2 orm reference:

"Why are many-to-many associations less common? Because frequently you want to associate additional attributes with an association, in which case you introduce an association class. Consequently, the direct many-to-many association disappears and is replaced by one-to-many/many-to-one associations between the 3 participating classes."

The association class is your third entity which you can use to create your search queries using JOINs for example.

In this case we have a pure many to many relationship. I built a form with an element of type entity which represents the work categories.

As this draws a multiselect control with the ID of the work categories as value on submitting the form, you have access to them in your search query. The key is to use "MEMBER OF" then:


Surprisingly for me, this creates a very similar query as I used for the one-to-many search query:

AND EXISTS (SELECT 1 FROM users_workcategories u2_ INNER JOIN user_work_category u3_ ON u2_.workcategory_id = u3_.id WHERE u2_.user_id = u0_.id AND u3_.id = ?) AND EXISTS (SELECT 1 FROM promoters_workcategories u2_ INNER JOIN user_work_category u3_ ON u2_.workcategory_id = u3_.id WHERE u2_.user_id = u0_.id AND u3_.id = ?)

Doctrine is aware of of the third relational table and uses it for JOINs to assemble the native SQL query that looks far more complicated than the DQL string. For people not familiar with advanced SQL queries, this is a live safer. On the other hand, SQL people surely need to be argued into this new approach.

Search form using symfony2 and doctrine2

This post is about creating a search form and using the doctrine2 query builder to dynamic built the query. You need an object you can bind the form values to. I used the symfony2 entity generator to build my heavy search form data holder really quickly. Let it create an empty repository class for you, which will hold the search function later. Use the symfony2 form generator to create a form based on the entity. As this entity won't be saved in the database the doctrine annotations are not needed. I kept them for later use and just removed the "Entity" annotation.

Next thing you do is writing a template for your form and to specify some validation rules. My controller action which receives the search POST looks like this:



We need to implement the findBySearchCriteria function in the empty repository class of the user entity. This function returns a query builder object which we can do everything we want with.

The following examples show what is possible with the doctrine2 query builder in the search function.

Many-to-one search

The user has a languages attribute which is of type Language. The entity Language has three attributes:

1. a language name (e.g. english, french, etc)
2. points that define how good the language is spoken by the user (10 very good, 1 very bad)
3. a back reference to the user

We want to find users that marked their english knowledge at least with 5 points and their french with at least 7 points. To accomplish this, we need to check, if the user has an language entry which fulfills the given criteria. We check, if such a language entry exists.

As "exists" in the query builder expects a dql subquery, we give it one.

"u" is the alias for the User class of the main query builder. The same method is used for the french language points.

The resulting DQL is:

SELECT u, l FROM Acme\UserBundle\Entity\User u WHERE (EXISTS( SELECT langE FROM AcmeUserBundle:Language langE WHERE langE.user = u AND langE.language = :langName AND langE.points BETWEEN 10 AND :points))

Nested AND / OR queries

Next requirement is that the zip code of the user entity has to lie in a given range of zip codes. The user who defines the search criteria is able to provide a list of zip code ranges. The format for a range is "xxxxx-xxxxx", so a zip code consists of five digits.

The zip code range search query needs to be built using OR. The user's zip code can lie in one range OR another. Native SQL could look like this:

WHERE (zipCode BETWEEN :zipCodeFrom1 AND :zipCodeTo1) OR (zipCode BETWEEN :zipCodeFrom2 AND :zipCodeTo2)

To combine this with the language query, the OR part has to be nested within an AND part:

WHERE ((zipCode BETWEEN :zipCodeFrom1 AND :zipCodeTo1) OR (zipCode BETWEEN :zipCodeFrom2 AND :zipCodeTo2)) AND (EXISTS( SELECT .... 

To accomplish the OR nesting you can use add() on the orx expression. The following example shows how to do that:


This results in a really nice nested query like this:

SELECT u FROM Acme\UserBundle\Entity\User u WHERE ((u.zipCode BETWEEN :zipFrom1 AND :zipTo1) OR (u.zipCode BETWEEN :zipFrom2 AND :zipTo2)) AND (EXISTS( SELECT langE FROM AcmeUserBundle:Language langE WHERE langE.user = u AND langE.language = :langName AND langE.points BETWEEN 10 AND :points))

Building nested queries is really easy. It took me some time to find out that I can use add() on the orx expression to dynamically add parts to it.

Notes on Zend SOAP webservice development

Developing SOAP webservices with the Zend Framework is very easy, but there are some things that aren't explicitly mentioned in the documentation. Hopefully this hints will help you to save time.

WSDL caching

While developing a webservice application do not forget to disable WSDL caching, e.g. in your php.ini configuration file.

soap.wsdl_cache_enabled=0

Set path to SOAP server in WSDL file

When you use a controller for your webservice and desire to have two actions, one for the automatically generated WSDL file and one for the server, use the setUri method of the Zend_Soap_AutoDiscover class to set the URI of the server.

$wsdl->setUri('http://example.localhost/index/soapserver');

Error handling

Error handling with exceptions is easy. I suggest to create your own Exception class first, it can be empty for now:

In your webservice class throw your new exception on invalid input for example.
On SOAP server side, you have to register your Exception class:


On Soap client side embed your request in a try - catch block:

Which for example outputs: 'SOAP ERROR: [Receiver] The product ID does not exist!'

The code

The webservice controller:


The webservice class:


The custom Exception class:

Google Chrome double request session problem

While working on an older PHP project based on a self made framework I noticed strange behaviour when running the project in Chrome. Everything worked fine in all other browsers, no matter, if it was production or development environment.

Each time I logged in the session ID cookie changed. The consequence was, that when trying to reload the page or open other restricted pages the system logged me out.

Even though everthing worked fine on other browsers, I tried to find the bug in the PHP code. I thought I missed something, set wrong headers which Chrome interpreted more strictly than other browsers or something similar. None of that was true. I debugged the code responsible for the session, made sure nothing is going to fall through the cracks. Code that was intended to run once on page execution ran twice, which lead me to the idea, that it's a forwarding/reloading issue.

I checked the $_SERVER array for header information, which differed between browsers. But there was nothing suspicious. REQUEST_URI then revealed the problem: While you would expect only one request to '/' Chrome requested '/favicon.ico' in a second request. This somehow caused a reload with a seperated cookie space, because the reloaded page couldn't read the session cookie and created a new one. I'm not sure what happened in detail but providing a favicon.ico file helped.

I'd like to emphasize that I didn't use a favicon in the HTML, perhaps there is a flag set in the server configuration which is ignored by other browsers.

Hope this helps someone, this took me some time to figure out.

Font problems with wkhtmltopdf

Recently I ran into problems with font kerning and font sizes when creating PDFs with wkhtmltopdf on different machines. I create PDFs on my ubuntu machine locally and on a FreeBSD server remotely.

Unfortunately font sizes differed a lot and from the beginning the kerning wasn't as I expected. The space after characters like 'v' is too small or too big, depending on the chosen font size. Sometimes it happens that two characters get rendered  one on the other. This is not acceptable.

I found out that this topic is already discussed in the wkhtml bugtracker. Users have different approaches to that problem, but none of the solutions is acceptable for me. Running an XServer is not possible and running xvfb as a wrapper to the tool sounds a lot slower than using wkhtmltopdf 'natively' and certainly is!

After experimenting with CSS modifications concerning font sizes, letter spacing, --zoom and --disable-smart-shrinking / --enable-smart-shrinking parameters to wkhtmltopdf I finally found a working solution. It's very easy to apply and has no disadvantages.

I need polish characters in my texts, so I downloaded high quality fonts from TeX Gyre Project. They provide their fonts in ODF format instead of the commonly used TTF format. I don't know, if the fonts of the Gyre Project solve the problem or if it is the newer ODF format that eliminates certain limitations of the TTF format when used on different operating systems.

In your external or internal stylesheet of your HTML file you'd like to convert, define a custom font face like this:


Don't forget to use the correct path to your font. Whenever you want to use it, use your new defined font family:


This eliminates all problems arising due to different font versions on different systems and the kerning bug in wkhtmltopdf version 0.10.0_beta5

Filling PDF forms using XFDF

Description

This article is about user data that can be provided in PDF form back to the user. Don't use old methods in the web which suggest to use FDF. FDF and the PHP FDF module are deprecated, don't support UTF-8 and you'll have only problems with it ;). 

One common use case is a web form that requires the user to input information for an application that has to be sent by snail mail for confirmation or to print the information in a user friendly form that is exchangeable.

The key is creating an XFDF file that holds all the user information connected to IDs of form input fields that are defined in a PDF file with empty inputs. The empty fields are then filled with the user information. The URL to the PDF file is defined in the XFDF file. If the user opens the XFDF file with the help of the Adobe Reader browser plugin, all happens in the user's browser window. An alternative for the user is downloading the XFDF file to open it later.

Advantages:

1. As filled PDF forms can't be saved (but can be printed), this is an alternative to keep the inputted data,
2. you don't need HTML to PDF converters. Create professional looking PDFs in any program you like

Disadvantages:

1. If the PDF file is available online, the user has to have an active internet connection whenever he or she opens the XFDF file, 
2. Adobe Reader asks, if it should fill out the PDF with possibly unsecure data. The PDF stays empty without user's confirmation.

One workaround for the 1st disadvantage is PDFToolkit. It allows to flatten a PDF file with the XFDF file to create a user ready downloadable PDF file, but the data is not editable anymore. The 2nd workaround is to download the empty PDF file to disc and change the URL in the XFDF file which is in XML format.

Creating PDF forms with OpenOffice

OpenOffice has a toolbar with form elements. You can find it in the View->Toolbars->Form Elements menu. Just place the elements you need in the document and export the whole thing as PDF. Don't forget  to check the "Create PDF Form" option. Please be aware of the IDs of the form elements in the document. You can set the ID by chosing "Properties" in the context menu of the element. To check the form elements afterwards you can use PDFToolkit:

pdftk test.pdf dump_data_fields
\--\-
FieldType: Text
FieldName: textelement1

FieldNameAlt:
FieldFlags: 0
FieldValue:
FieldValueDefault:
FieldJustification: Left

Generating the XFDF file

The specification I used can be found on Adobe Website. This is the PHP code to create an XFDF file out of an array of user input data fields.


And this is how to use it:


As you can see, only the XFDF file is outputted. The Adobe Reader plugin parses it and tries to download the corresponding PDF file to fill in the user data.

Dynamic HTML to PDF conversion

There are a lot of libraries out there which allow you to create PDFs from HTML documents. Some of them even allow to use external stylesheets, others allow inline styles only. They have one disadvantage in common: the PDF never looks like your original HTML file. Other limitations include i.e.:

• No support for floating,
• no support for ordered lists,
• no support for nested tables,
• in general table support is weak for all libraries I tested,
• only a subset of CSS styles is supported,
• dynamic content placement is very difficult,
• problems on page breaks,
• etc.

The easiest way is to just print the HTML page to a PDF file. The generated files are nearly 100% as you expect them to be. The best solution I found to accomplish this is wkhtmltopdf. Ask your webhosting provider to install this tool, if you don't have the possiblity to install it by yourself. I heard of people that managed it to install wkhtmltopdf in their home directory successfully.

wkhtmltopdf uses QT libraries. These libraries include the Webkit rendering engine for Apple Safari Browsers. So you can expect to have all CSS and HTML features working that Safari currently supports. You have to compile the author's patched QT libraries to have full functionality without the need of using a running XServer. Alternatively use the static executables.

You can find a PHP Class you can use with wkhtmltopdf. It's pretty self explaining and the example application should help you creating your first PDF file. I create PDF files by rendering an HTML file out of my PHP application, saving it to a writable directory with other necessary external files like images, js and css and executing the wkhtmltopdf converter.

It's a great tool and creates small PDF files. I'm missing security features like password protection, though.