Refactoring your legacy code - Part One: In the beginning there was….
As promised Tomas and me will try to give a rough overview what you need to consider when you plan to refactor your old (legacy) applications. It won’t be a detailed guideline nor very much sorted. More or less our toughts and experiences over the last years.
If you want a more detailed (including sourcecode level) introduction you should attend our workshop on phpconference. As I will run this workshop together with two of the best known PHP developers ( Johann Peter Hartmann and Thorsten Rinne) you can expect much more details.
For sure I will upload the presentation after the IPC - but not before 
Now - refactoring. Hmm… Refactoring means per definition modifying (cleaning up) it without changing its behavior. But how do you make sure that you won’t change funtionality if there are no tests ? This means - you only can refactor when there are tests. And this is where the problem starts - no tests, or ?
And there we go. Let’s assume you do have a larger project - growed over the years. And now you are in the lucky situation that your boss agree’s with you about the need of tests. But where to start ? The team has no experience in writing tests. Nor Test Driven Development what is even worse.
You will need to invest some time into the team so that they learn about writing tests. And you will experience that knowing and understanding are two different things. From my personal opinion there is only one thing which can be compared. The difference between procedural and object oriented programming. Strange example ? No.
Let me explain. It is very easy to get a book and learn a bit about classes, objects etc. But only after using it for a while you will deeply understand the concept of OOP and get the sense. Same is for TDD - writing a test is more or less a matter of minutes. Or let’s say hours if its your first. But understanding why tests are important and how to use them. This needs time. And you will have to invest this time.
After I pushed the team into this direction ( and hey - this was not so easy as most developers tend to be very conservative ) they did the tests - simply because the boss said so. But only a few weeks one Developer told me “Hey! Tests are cool! I found a bug I would have never found before”. This is the point where you need to get your team to.
So much about the theory. But how about the tests. Where to start? The answer is easy. You need to start with the worst, nastiest and largest file you can find in your project. I know I know. They want to start with the easier files, where tests are done fast and you see some progress. Developers will really fear this file. But what will happen if you start with the easy ones? Simply - you will see some progress and get the false feeling that things are alright. Your team still won’t understand the tests fully - and they leave the nasty piece of code what to the end. You have them to force to adress their demon in the beginning. This will take a while. But then you can be sure that the rest will be a piece of cake. Otherwise you will move the risky part to the end of the refactoring period - and this is not a good idea.
You might think “this guy is nuts where are the problems in writing tests…”? Hey! We are talking about old, grown ( == spaghetti ) code. This code is interconnected massively. Globals, mixed classes, wild calls between modules and objects etc. It is very difficult to write tests for this.
Actually while writing tests you will note that you need to refactor the code. You simply have to. As otherwise you can’t write a test. Or - let’s be honest - you can. If you write mocks and stubs which are triple the size of your code. This is exactly what unexperienced TDD developers will do. If you see this - kick them. And then again just because it feels good
Each tests leads to refactoring. Like the gordian knot you will start to pull the worm and you will pull, pull and pull. And after some while refactoring is done and the first test can be written. Ok just kidding. But there is some truth in this statement. You need to refactor the code first - then write the test. No large stubs.
And this will be a lot of work - especially in the beginning as everything is interconnected. And these dependencies need to be addressed and solved. First. Seperate the model and the view. In the View leave no logic - just I/O. And then test the model. Some people even test the View with unit tests. I don’t. I do prefer testing the model, having small views without logic and leave the rest to the selenium tests.
Nasty. Takes long. Much work. But believe me - there is no way around.
While reading you might get the idea that you will get away with acceptance tests ( selenium ). Do you think this could do the deal? The idea is thrilling. If you will write selenium tests for your application fully - then refactor - and you did not break any tests you could be sure that you did not change any functionality. Nice idea? Won’t work. Sorry.
Remember - I was talking about a larger grown application. This one will have tons of functionality. This means 100s or 1000s of tests. And these tests can last long. I know it. I went down this road - and it was a oneway road. We ended with one full test running 10 hours. This is not Test Driven Development. For doing that you need instant - instant - feedback. Not on the next day. Nobody can work like this.
Let’s continue in Part Two.
Refactoring legacy Code ?
It has been a while since my last posting - sorry about that. Actually I was - and am - really busy with the new milestone of swoodoo.com which we will (hopefully) release soon.
We did refactor nearly the whole system - starting from the database layer, over to the API and finally the GUI. Everything will be brand new - or at least massivly refactored.
As we do agile development since many years we are quite experienced with managing such large refactoring sessions. More or less the same happend in OXID eSales - the new eShop Version was refactored massively. Under the hood (and not only there) many things changed.
And how about you ? Over the last years I heard many developers, managers and companies stating that they would love to change/refactor their big ball of mud - but they won’t as they fear to fail. And in deed - the risk to fail is definitly there.
On the upcoming php Conference near Frankfurt Johann Peter Hartmann, Thorsten Rinne and me will run a full day workshop where we try to show you, where the devil in the details is creeping out of his hole.
For the ones who can’t or don’t want to attend the php Conference (which is a huge mistake but hey - it’s your decision) my friend Tomas Liubinas (Senior Developer at OXID eSales) and myself will try to give you a brief overview over the pitfalls and learnings we had in refactoring OXID eShop.
This will be a small series of articles - as otherwise it would be a monster posting 
Check back soon.
Drogenscreening in der agilen Softwareentwicklung?
Der looser ist mal wieder gut immer drauf und hat mit einem Geistesblitz einen sehr interessanten Zusammenhang hergestellt:
Mir ist in meinem aktuellen Projekt aufgefallen, dass eine Korrelation zwischen Drogen-Konsum-Menge und verbleibender Projekt-Zeit bestehen könnte.
…
Das Interessante an diesem Modell ist schließlich, dass es einen sehr guten Projekt-Status liefern wird, weil sich der Projekt-Zustand am Projekt-Team messen lässt. Man denke an Projekte, die daran gescheitert sind, weil es keine Club-Mate mehr gab. Der Einsatz eines integrated continuous drug screenings kann hier also aktiv helfen.
…
Mehr davon? Lohnt sich! Hier.
PHP Unit Database fixtures “the ruby way”
More or less everybody who does test driven development comes to the point where one realizes that the preparation of the database before each test is vital - for good results and for your mental health. If you don’t do this properly then you will find yourself at some day searching through hundreds ( or thousands ) of tests to find the one which screws the database so that subsequent tests fail - even tough they are developed perfect.
There are many ways how to solve this issue - let’s have a look:
- Manually
Each test inserts, update’s and delete’s data on his own. You have to agree in your team that the table’s are in the same state at the end of the test like it was before. I think this is the most common case - and I can tell you - DON’T DO THIS. It will lead you into troubles as long as you are not developing alone. There is always a “smartass” in your team who does not fully cleanup - or even worse delete contents which are expected to be in the table. You will have chaos - and you will end up searching for exactly these tests. No fun, much work - slows you down a lot. So - let’s skip this.
- Use DBUnit
Mike Lively Jr. ported the Java DbUnit Extension into PHPUnit. Really nice work. It does the job and is the right tool to be used. There is already a lot of information around concerning DBUnit - see Mike’s blog or Sebastian Bergmann’s Slides. As I already said - DBUnit is fine, doing the job. But this blog wouldn’t be called “Frontalaufprall” ( frontal impact ) if I would not try to do thing’s different ( == better ). And this is where it get’s interesting. I want to show you a more unusual way to prepare your fixtures - the “ruby” way.
Ok ok - we are all PHP evangelists - but still - not everything in Ruby is bad. There are a lot of things we can integrate and learn from. If you can’t beat them - join them
- Let’s have a closer look
Ruby uses YAML. YAML Ain’t Markup Language, it’s a readable, friendly language for storing lists, dictionaries, text, numerics and more.
And this is already the main difference to DBUnit where you usually write XML. This was the reason why we decided in our swoodoo Team to give YAML a chance - we were simply to lazy to deal with all these “<>” thingies.
For using YAML you need syck. Syck is a parser for YAML and cares about reading and writing. Syck is fast - the author claims
Do not disturb Syck because it is so focused on the task at hand that it will slay you mortally if you get in its way.
Actually I never benched it so I simply believe that it’s fast. Would be interesting to let it run vs. DbUnit one day - if I find time I will do this. ( very unlikely… )
As Symfony uses Syck - you can find really good installation description here and I can focus on the integration into PHPUnit.
Successful installed? Now we can talk php.
Each Testfile should have his own fixture file - best practise is that you create a “fixture” tree parallel to your test directory where you keep these files. The contents could look like this:
1:
APIKEY: ‘<?php echo $something ?>’
CUC: EZY
TRACKING_ID: 2
AFFILIATE_ID: 4545
MISC:
2:
APIKEY: ”
CUC: EXPD
TRACKING_ID: 3
AFFILIATE_ID: 45456
MISC:
Loading is done via
public static function create($fileName) { $fileName = 'Fixtures'.DIRECTORY_SEPARATOR.$fileName; ob_start(); include $fileName; $fileContents = ob_get_contents(); ob_clean(); $yamlData = syck_load($fileContents); return $yamlData; }
Do not be surprised about the freaky “loading” code - you can use “file_get_contents” if you want - but then you loose the possibilty to execute php code. And as you can see in my example sometimes it’s really good to dynamically create data.
Now we need to insert the data into our database.
public static function load($fixtures, $tableName) { if (is_array($fixtures) && count($fixtures)) { foreach ($fixtures as $fixture) { if (is_array($fixture) && is_array(current($fixture))) { Fixtures::load($fixture, $tableName); } $fields = array_keys($fixture); $statement = "INSERT INTO $tableName (" . implode(', ', $fields) . ") VALUES (:" . implode(", :", $fields) . ")"; $stmt = self::$_db->prepare($statement); if (count($fixture)) { foreach ($fixture as $key => $value ) { $stmt->bindValue(':'.$key, $value); } } $stmt->execute(); self::$_usedTables[$tableName] = $tableName; } } }
Where we assume that self::$_db is a valid PDO object - and self::$_usedTables is a simple array where we do store which tables we touched for proper rollback. More or less - this is it. In your setup Method of you test you load the data - and in tear down you reset it via:
if (!empty(self::$_usedTables)) { foreach (array_reverse(self::$_usedTables) as $tableName) { self::$_db->execute("TRUNCATE TABLE $tableName"); } }
All this should be put into a helper class - in our case we called it “Fixtures”. The loading and the resetting we did implement into a baseclass of all our tests - this means when we do write a test we can very comfortably load by calling:
$this->_fixtures = Fixtures::createAndLoad(’PageGeneratorTest’ . DIRECTORY_SEPARATOR . ‘updatepagegen.yml’, ‘page_generator’);
and this is it. Every Test starts with a fresh database and after each test it’s truncated again. No troubles with corrupted data. And it’s fast.
Note the ” $this->_fixtures = ” - by passing the loaded data back we can test against the fixture - not some static data. So when you change the fixture ( e.g. as you extend the tables ) you do not have to touch your tests:
$this->assertEquals( $this->_fixtures[4]['ID'], $ret[0]['id']);
That’s it. As I hopefully could show you - using YAML together with PHPUnit or any other Testtool works fine in PHP.
I see two main advantages of YAML over DBUnit:
- no hazzle with XML format
- fast, easy and lightweight
So if you decide for any reason not to go the “normal” way and use DBUnit - then you will be fine using YAML with syck. And if it’s just to cheer your team up that they can use at least some small parts from ruby
P.S. I would be __VERY__ happy if someone could let me how I can avoid the translation of extended chars in the sourcecode examples ( & => & ) …. I did not find an easy way yet to avoid this.
„Ein agiler Prozess wie eXtreme Programming hat einen deutlich messbaren Einfluss auf die Qualität einer Software.“
Im aktuellen PHP Magazin ( Special Security und QA ) ist ein Artikel von mir erschienen in dem ich mich bemühe in aller gebotenen Kürze den Einfluss von eXtreme Programming auf die Software Qualität zu beleuchten.
Zugegebenermassen - es ist mir schwer gefallen. Fünf Seiten sind zuviel um das ganze nur kurz zu beleuchten und zu wenig um ernsthaft in die Materie einzusteigen. Nichtsdestotrotz hoffe ich das es mir gelungen ist, bei dem einen oder anderen Lust auf neue Prozesse zu wecken und evtl. ein paar Denkanstösse zu liefern
Qualität ist ein wichtiges Merkmal einer Software und hat einen erheblichen Einfluss auf ihren Erfolg oder Misserfolg. Neben den klassischen Methoden zur Qualitätssicherung hat auch der Software Entwicklungsprozess einen entscheidenden Einfluss auf die Häufigkeit von Bugs oder auf Missverständnisse zwischen Entwickler und Kunden – welche meist Grund für inhaltliche Fehler sind. Welche Auswirkungen auf die Qualität hat nun ein agiler Prozess wie eXtreme Programming?
Also - PHP Magazin besorgen und lesen.
Unit Testing your Javascript Code.
As it’s been a while since my last “useful” posting I will bite the bullet now and therefore I will try to point you to a very easily overseen part of unit testing - the Java Script part.
Unit Test and Javascript ? Oh yes - we are living in the Web 2.0 (or even 3.0?) century and most applications do have huge parts which are written in Javascript. And for sure you need to test these part’s also - especially if you are following the eXtreme way of programming.
There are some tools around doing this, most popular is probably jsunit but today I want to focus on Crosscheck - an open source, java based test framework for your javascript code. Crosscheck’s huge advantage is that you do not need any browser - it simulates them so it can be run from commandline. The upside is - it works really nice, integrates smooth with Cruisecontrol and tests on different browsers. But for sure there is a downside - and this is - it works without the browsers and simulates them.
Unfortunatly simulating a browser can never be perfect, so you will have tests which will give wrong results - either the tests claim that it should run, but in real world it won’t or vice versa. Also some popular libraries like YUI are not yet supported and throw errors - but AFAIK thy are working on it - let’s hope the best.
The best place for all infos around Crosscheck is the developer site.
Now let’s start from the scratch - point your browser to ‘thefrontside’ and download the file. Make sure that you download the compiled version and not the source (*-sc*) version as these won’t work for you.
Unzip, and copy the file ‘crosscheck.jar” where ever you want to have it.
Now test if it’s working:
crosscheck$ java -jar crosscheck.jar
must print something like
Please specify at least one test file or directory.
Usage: java -jar crosscheck.jar [options] [test-file | test-directory]*
…
If not - install the latest JRE on your machine, and make sure that JAVA_HOME is set correct and the “java” binary is in your PATH.
For easier usage on the commandline and later in cruisecontrol we should create a shell script to wrap this call - so let’s do it. It could look like this:
java -jar crosscheck/crosscheck.jar $@
Also don’t forget to chmod +x the .sh file.
Creating our first Testfile is quite easy. Create a directory “js” in your “tests” directory whereever you also have your unit tests - simply create the directory parallel to “unit” directory and there we add a file called “demo.jst” with the following content:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | crosscheck.addTest({ setup: function() { //crosscheck.load("../out/js_source/filter.js"); }, /** dummy test example */ "test if something does something else": function () { assertEquals("0,- €", "0,- €"); } }) |
Now let’s have a look if it works:
$ crosscheck.sh demo.jst
And you should get something like this:
Running tests in environment: Mozilla 1.7 (Firefox 1.0)
0 tests: 0 Ok, 0 failed, 0 errored.
Running tests in environment: Mozilla 1.8 (Firefox 1.5)
0 tests: 0 Ok, 0 failed, 0 errored.
Running tests in environment: Internet Explorer 6
0 tests: 0 Ok, 0 failed, 0 errored.
So far so good, how does the integration with cruisecontrol now work ? Actually very very simple - this is why I have chosen to use crosscheck - it can export the junit format. Running crosscheck with -xml command produces JUnit files:
crosscheck.sh -xml=report js/
So this means you have to update your build.xml and add a new section for testing jScript - it could look like this:
<target name=”testjs” description=”Run the Javascript tests”>
<exec dir=”${project.srcdir}/tests” executable=”crosscheck.sh” failonerror=”false”>
<arg line=”-xml=${project.logdir}” />
</exec>
</target>
or whatever your config.xml / build.xml philosophie is.
I hope I made mayself clear - after reading this there should be no excuse for NOT testing Javascript.
And last but not least a big THANK YOU to the guys at Frontside, Charles Lowell and Jason Wadsworth for the good work!
Debugging Unit Tests ?
It might sound trivial and the solution is more than easy - but as I am frequently asked how todo this, here is the simple answer.
Create a PHP file in the directory where you start your Unittests usually with this content:
1 2 3 4 5 6 7 8 9 10 11 12 13 | $unitTest = '...Your Unittest name - e.g. unit_mytest.php or Alltests.php etc...'; $_SERVER['argv'] = array( '/...path to your PHPUnit.../PHPUnit/TextUI/Command.php', $unitTest ); echo "<pre>"; include( '/...path to your PHPUnit.../PHPUnit/TextUI/Command.php' ); echo "</pre>"; ... |
Point your browser to this file, et voila - Unit test in the Browser.
Ok - this solution is very easy and silly - but it works very well. And at least I got the impression that there are enough people around who do not know how to debug unit tests…
Testing Buildix
So as promised I finally found some time to test Buildix from Thoughtworks.
- Installed a fresh Ubuntu Version
- Installed Buildix via apt-get as described.
Unfortunatley it did not work as expected - after a short search in the forums I found the correct hint here.
Simply set execute and write permission of the folder /usr/share/buildix for all other users and reboot the system. That’s it.
Ah Mingle - I was always interested in this software - so let’s give it a try. After filling in the settings for SMTP I created my first project - and - nothing happens. Server hangs, load is very high. Too bad - I simply lack the time to contact the support to figure out what happens - therefore I had to remove Mingle…
But - all the rest works great - I am amazed. I think this is the most easy way to get your continious integration server up and running.
Only User Management seem to lack some functionality. Right now everybody can register himself for write access to svn what is probably not the very best idea in the world. But it seems that they are working on it.
Let’s see how I will come along with Buildix…
Tales from the eXtreme Side - IPC 07 is over
So finally after 5 days I survived IPC and made my way home. As always it was a real pleasure to meet “la famiglia” - I learned a lot, hopefully could teach other people something and definitly had too much beer. The real reason why the IPC is so short is probably that otherwise 90% of PHP community would have to detox from alcohol
So - the title from my last talk could have also been the title of the whole IPC.
For those of you who are interested in the slides - you can download them directly here: tales-from-the-extreme-side_ipc_07.pdf
or if you interested in the PPT you might be able to fetch them via http://phpconference.com/
Agile Development for Beginners at IPC 07 in Frankfurt
Yesterday I did my workshop about Agile Development for Beginners - the slides will be available via the official phpconference website - and you can download them directly here.
agile-development-for-beginners_ipc_07.pdf
I think the workshop went quite well - and I found already some interesting feedback here - check it out.