Backup your SVN/CVS Repository…
Did you ever think about what happens if your office/server location is on fire? We all do backup (hopefully) our data frequently - but let’s be honest - most of the time we simply store the data on some other machine located in the same place. In some cases this is not enough - therefore you must store your data in some other physical location.
In my past companies we solved this usually by storing a CD somewhere in a bank case - every 6 months…. Luckily nothing bad happend to our Servers in the past - but you can never know. And I’ve seen this happening recently (no I won’t tell you where) and it’s hell of a work to merge all the CVS copies on the developer laptops back - and for sure some history was lost.
Here at swoodoo we do have our main SVN Server running in a VMWare Image - and this Image is fully backed up frequently - in the same location. Therefore I was searching for a easy solution - and do not underestimate the word “easy”! I’ve seen so many cool backup solutions which slowly disappeared over the time as people did not like to use it - or simply did not understand fully how they must use them.
Fortunately there is a perfect solution for lazy people like me. The “Gigabank“. It’s a WORM storage over the Internet somewhere locked away in switzerland. This data is safe and won’t be deleted or removed as this is technically impossible.
Background infos:
Your files are protected in the Gigabank using the unrivaled, patent pending FAST LTA WORM technology. WORM stands for write once, read many and that means your files cannot be deleted or altered accidentally by FAST LTA or by yourself. All files are also secured against attacks from viruses or other kind of malware and hackers.
…
FAST LTA has no insight into your files, and we make sure no-one else has access by storing all data in Switzerland, in modern data centers run by Swisscom…
…
Gigabank uses a combination of AES 256 encryption and SSL secure data transmission to ensure the safety of data.When a client has to transfer data to the LTA-CH1, it connects using a secure SSL (1024bit RSA key exchange, 128 bit RC4 stream cipher and 160 bit SHA-1 integrity checking) connection to transfer the data. Signed SSL certificates are used to verify server integrity.
Gigabank uses AES 256 end-to-end encryption. All data is encrypted before it leaves the client system. The data remains continuously encrypted while in storage and during over-the-wire transmissions. Decryption following a restore only occurs on the client system when the encryption key holder enters the encryption key. The encryption key is only known by the client.It will never be transmitted over the Internet and it is never stored on any FAST server.
For more info about the technical details point your browser here.
Enough blabla let’s see how it works.
Create your account here - 1 GByte is for free, for more you need to pay. And please note - nothing is deleted - so each backup will add up to the quota.
Now let’s setup a short bash script which dump’s the repo, zip it and then let’s upload the file automatically. Zipping is VERY important otherwise you will have a massive amount of files in your storage which might screw up at least the browser interface 
Our script could look like this:
#!/bin/sh
svnadmin dump /var/svn/swoodoo > /backup/swoodoo-goa.dump &&
tar -czf /home/swoodoo/svn-backup/swoodoo-goa-`date +%F`.tgz /backup/swoodoo-goa.dump &&
md5sum /home/swoodoo/svn-backup/swoodoo-goa-`date +%F`.tgz > /home/swoodoo/svn-backup/swoodoo-goa-`date +%F`.md5 &&
gbcopy –source /home/swoodoo/svn-backup/swoodoo-goa-`date +%F`.tgz -u $1 -p $2 &&
gbcopy –source /home/swoodoo/svn-backup/swoodoo-goa-`date +%F`.md5 -u $1 -p $2 &&
rm /home/swoodoo/svn-backup/swoodoo-goa-`date +%F`.tgz &&
rm /backup/swoodoo-goa.dump
So what do we do here ? First dump, then zip (tar) it, set a good filename and upload using gbcopy which is the linux commandline tool from Gigabank. Additionally for security reasons we store the md5 key separate - and finally, cleanup.
That’s it. Add a cron job to be executed every 14days or so - and you will sleep better
PHP4 ist tot.
Johannes ist der aktuelle Release Master von PHP.
Johannes: PHP 4 ist tot, tot und nochmal tot!
Johannes: soll ruhig jeder verstehen, PHP 4 ist tot, offiziell besteht bis zum 8.8. noch ein Zeitfenster für kritische sicherheitsfixes aber das müssten dann schon triviale root-exploits sein oder so
Also bitte - wer seine Software auf öffentlichen Servern noch unter PHP4 betreibt handelt grob fahrlässig.
PHP Unit Database fixtures “the ruby way”
More or less everybody who does test driven development comes to the point where one realizes that the preparation of the database before each test is vital - for good results and for your mental health. If you don’t do this properly then you will find yourself at some day searching through hundreds ( or thousands ) of tests to find the one which screws the database so that subsequent tests fail - even tough they are developed perfect.
There are many ways how to solve this issue - let’s have a look:
- Manually
Each test inserts, update’s and delete’s data on his own. You have to agree in your team that the table’s are in the same state at the end of the test like it was before. I think this is the most common case - and I can tell you - DON’T DO THIS. It will lead you into troubles as long as you are not developing alone. There is always a “smartass” in your team who does not fully cleanup - or even worse delete contents which are expected to be in the table. You will have chaos - and you will end up searching for exactly these tests. No fun, much work - slows you down a lot. So - let’s skip this.
- Use DBUnit
Mike Lively Jr. ported the Java DbUnit Extension into PHPUnit. Really nice work. It does the job and is the right tool to be used. There is already a lot of information around concerning DBUnit - see Mike’s blog or Sebastian Bergmann’s Slides. As I already said - DBUnit is fine, doing the job. But this blog wouldn’t be called “Frontalaufprall” ( frontal impact ) if I would not try to do thing’s different ( == better ). And this is where it get’s interesting. I want to show you a more unusual way to prepare your fixtures - the “ruby” way.
Ok ok - we are all PHP evangelists - but still - not everything in Ruby is bad. There are a lot of things we can integrate and learn from. If you can’t beat them - join them 
- Let’s have a closer look
Ruby uses YAML. YAML Ain’t Markup Language, it’s a readable, friendly language for storing lists, dictionaries, text, numerics and more.
And this is already the main difference to DBUnit where you usually write XML. This was the reason why we decided in our swoodoo Team to give YAML a chance - we were simply to lazy to deal with all these “<>” thingies.
For using YAML you need syck. Syck is a parser for YAML and cares about reading and writing. Syck is fast - the author claims
Do not disturb Syck because it is so focused on the task at hand that it will slay you mortally if you get in its way.
Actually I never benched it so I simply believe that it’s fast. Would be interesting to let it run vs. DbUnit one day - if I find time I will do this. ( very unlikely… )
As Symfony uses Syck - you can find really good installation description here and I can focus on the integration into PHPUnit.
Successful installed? Now we can talk php.
Each Testfile should have his own fixture file - best practise is that you create a “fixture” tree parallel to your test directory where you keep these files. The contents could look like this:
1:
APIKEY: ‘<?php echo $something ?>’
CUC: EZY
TRACKING_ID: 2
AFFILIATE_ID: 4545
MISC:
2:
APIKEY: ”
CUC: EXPD
TRACKING_ID: 3
AFFILIATE_ID: 45456
MISC:
Loading is done via
public static function create($fileName) { $fileName = 'Fixtures'.DIRECTORY_SEPARATOR.$fileName; ob_start(); include $fileName; $fileContents = ob_get_contents(); ob_clean(); $yamlData = syck_load($fileContents); return $yamlData; }
Do not be surprised about the freaky “loading” code - you can use “file_get_contents” if you want - but then you loose the possibilty to execute php code. And as you can see in my example sometimes it’s really good to dynamically create data.
Now we need to insert the data into our database.
public static function load($fixtures, $tableName) { if (is_array($fixtures) && count($fixtures)) { foreach ($fixtures as $fixture) { if (is_array($fixture) && is_array(current($fixture))) { Fixtures::load($fixture, $tableName); } $fields = array_keys($fixture); $statement = "INSERT INTO $tableName (" . implode(', ', $fields) . ") VALUES (:" . implode(", :", $fields) . ")"; $stmt = self::$_db->prepare($statement); if (count($fixture)) { foreach ($fixture as $key => $value ) { $stmt->bindValue(':'.$key, $value); } } $stmt->execute(); self::$_usedTables[$tableName] = $tableName; } } }
Where we assume that self::$_db is a valid PDO object - and self::$_usedTables is a simple array where we do store which tables we touched for proper rollback. More or less - this is it. In your setup Method of you test you load the data - and in tear down you reset it via:
if (!empty(self::$_usedTables)) { foreach (array_reverse(self::$_usedTables) as $tableName) { self::$_db->execute("TRUNCATE TABLE $tableName"); } }
All this should be put into a helper class - in our case we called it “Fixtures”. The loading and the resetting we did implement into a baseclass of all our tests - this means when we do write a test we can very comfortably load by calling:
$this->_fixtures = Fixtures::createAndLoad(’PageGeneratorTest’ . DIRECTORY_SEPARATOR . ‘updatepagegen.yml’, ‘page_generator’);
and this is it. Every Test starts with a fresh database and after each test it’s truncated again. No troubles with corrupted data. And it’s fast.
Note the ” $this->_fixtures = ” - by passing the loaded data back we can test against the fixture - not some static data. So when you change the fixture ( e.g. as you extend the tables ) you do not have to touch your tests:
$this->assertEquals( $this->_fixtures[4]['ID'], $ret[0]['id']);
That’s it. As I hopefully could show you - using YAML together with PHPUnit or any other Testtool works fine in PHP.
I see two main advantages of YAML over DBUnit:
- no hazzle with XML format
- fast, easy and lightweight
So if you decide for any reason not to go the “normal” way and use DBUnit - then you will be fine using YAML with syck. And if it’s just to cheer your team up that they can use at least some small parts from ruby
P.S. I would be __VERY__ happy if someone could let me how I can avoid the translation of extended chars in the sourcecode examples ( & => & ) …. I did not find an easy way yet to avoid this.
„Ein agiler Prozess wie eXtreme Programming hat einen deutlich messbaren Einfluss auf die Qualität einer Software.“
Im aktuellen PHP Magazin ( Special Security und QA ) ist ein Artikel von mir erschienen in dem ich mich bemühe in aller gebotenen Kürze den Einfluss von eXtreme Programming auf die Software Qualität zu beleuchten.
Zugegebenermassen - es ist mir schwer gefallen. Fünf Seiten sind zuviel um das ganze nur kurz zu beleuchten und zu wenig um ernsthaft in die Materie einzusteigen. Nichtsdestotrotz hoffe ich das es mir gelungen ist, bei dem einen oder anderen Lust auf neue Prozesse zu wecken und evtl. ein paar Denkanstösse zu liefern
Qualität ist ein wichtiges Merkmal einer Software und hat einen erheblichen Einfluss auf ihren Erfolg oder Misserfolg. Neben den klassischen Methoden zur Qualitätssicherung hat auch der Software Entwicklungsprozess einen entscheidenden Einfluss auf die Häufigkeit von Bugs oder auf Missverständnisse zwischen Entwickler und Kunden – welche meist Grund für inhaltliche Fehler sind. Welche Auswirkungen auf die Qualität hat nun ein agiler Prozess wie eXtreme Programming?
Also - PHP Magazin besorgen und lesen.
Unit Testing your Javascript Code.
As it’s been a while since my last “useful” posting I will bite the bullet now and therefore I will try to point you to a very easily overseen part of unit testing - the Java Script part.
Unit Test and Javascript ? Oh yes - we are living in the Web 2.0 (or even 3.0?) century and most applications do have huge parts which are written in Javascript. And for sure you need to test these part’s also - especially if you are following the eXtreme way of programming.
There are some tools around doing this, most popular is probably jsunit but today I want to focus on Crosscheck - an open source, java based test framework for your javascript code. Crosscheck’s huge advantage is that you do not need any browser - it simulates them so it can be run from commandline. The upside is - it works really nice, integrates smooth with Cruisecontrol and tests on different browsers. But for sure there is a downside - and this is - it works without the browsers and simulates them.
Unfortunatly simulating a browser can never be perfect, so you will have tests which will give wrong results - either the tests claim that it should run, but in real world it won’t or vice versa. Also some popular libraries like YUI are not yet supported and throw errors - but AFAIK thy are working on it - let’s hope the best.
The best place for all infos around Crosscheck is the developer site.
Now let’s start from the scratch - point your browser to ‘thefrontside’ and download the file. Make sure that you download the compiled version and not the source (*-sc*) version as these won’t work for you.
Unzip, and copy the file ‘crosscheck.jar” where ever you want to have it.
Now test if it’s working:
crosscheck$ java -jar crosscheck.jar
must print something like
Please specify at least one test file or directory.
Usage: java -jar crosscheck.jar [options] [test-file | test-directory]*
…
If not - install the latest JRE on your machine, and make sure that JAVA_HOME is set correct and the “java” binary is in your PATH.
For easier usage on the commandline and later in cruisecontrol we should create a shell script to wrap this call - so let’s do it. It could look like this:
java -jar crosscheck/crosscheck.jar $@
Also don’t forget to chmod +x the .sh file.
Creating our first Testfile is quite easy. Create a directory “js” in your “tests” directory whereever you also have your unit tests - simply create the directory parallel to “unit” directory and there we add a file called “demo.jst” with the following content:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | crosscheck.addTest({ setup: function() { //crosscheck.load("../out/js_source/filter.js"); }, /** dummy test example */ "test if something does something else": function () { assertEquals("0,- €", "0,- €"); } }) |
Now let’s have a look if it works:
$ crosscheck.sh demo.jst
And you should get something like this:
Running tests in environment: Mozilla 1.7 (Firefox 1.0)
0 tests: 0 Ok, 0 failed, 0 errored.
Running tests in environment: Mozilla 1.8 (Firefox 1.5)
0 tests: 0 Ok, 0 failed, 0 errored.
Running tests in environment: Internet Explorer 6
0 tests: 0 Ok, 0 failed, 0 errored.
So far so good, how does the integration with cruisecontrol now work ? Actually very very simple - this is why I have chosen to use crosscheck - it can export the junit format. Running crosscheck with -xml command produces JUnit files:
crosscheck.sh -xml=report js/
So this means you have to update your build.xml and add a new section for testing jScript - it could look like this:
<target name=”testjs” description=”Run the Javascript tests”>
<exec dir=”${project.srcdir}/tests” executable=”crosscheck.sh” failonerror=”false”>
<arg line=”-xml=${project.logdir}” />
</exec>
</target>
or whatever your config.xml / build.xml philosophie is.
I hope I made mayself clear - after reading this there should be no excuse for NOT testing Javascript.
And last but not least a big THANK YOU to the guys at Frontside, Charles Lowell and Jason Wadsworth for the good work!
Artikel im PHPMagazin über die Entstehung von Swoodoo
In der aktuellen Ausgabe des PHP Magazins ist ein Artikel von mir über die Historie und Entstehung von Swoodoo inklusiver lustiger (und nicht so lustiger) technischer Problem. Wer sich für das “crawlen” bzw. “screenscraping” wie es korrekt heisst interessiert der sollte das lesen.
Von einem, der auszog, das Crawlen zu lernen.
Eine Flugsuchmaschine wie SwooDoo zu entwickeln, erfordert sehr viel Arbeit und Know-how. Wie so oft liegt die Tücke im Detail – eine „Deep-Web-Suche“ sieht auf den ersten Blick recht einfach zu realisieren aus – erst während der Entwicklung zeigen sich die Probleme und Herausforderungen. Gerade die Reise- industrie bietet etliche Fallstricke, die man einzeln angehen und lösen muss.
Debugging Unit Tests ?
It might sound trivial and the solution is more than easy - but as I am frequently asked how todo this, here is the simple answer.
Create a PHP file in the directory where you start your Unittests usually with this content:
1 2 3 4 5 6 7 8 9 10 11 12 13 | $unitTest = '...Your Unittest name - e.g. unit_mytest.php or Alltests.php etc...'; $_SERVER['argv'] = array( '/...path to your PHPUnit.../PHPUnit/TextUI/Command.php', $unitTest ); echo "<pre>"; include( '/...path to your PHPUnit.../PHPUnit/TextUI/Command.php' ); echo "</pre>"; ... |
Point your browser to this file, et voila - Unit test in the Browser.
Ok - this solution is very easy and silly - but it works very well. And at least I got the impression that there are enough people around who do not know how to debug unit tests…
Testing protected Methods in Unit Tests
As a followup to my talks on the IPC 07 in Frankfurt here the improved version including:
- support for parameter in constrcutors ( via reflection )
- small bug fix avoiding autoload to conflict
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | /** * Create proxy of given class. Proxy allows to test of protected class methods * @param string $superClassName * @param array|null $constructorParams parameters for contructor * @return object */ function getProxy($superClassName, array $params = null) { $proxyClassName = "{$superClassName}Proxy"; if (!class_exists($proxyClassName, false)) { $class = <<<class> class $proxyClassName extends $superClassName { public function __call($function, $args) { $function = str_replace('protected_', '_', $function); return call_user_func_array(array(&$this, $function), $args); } } CLASS; eval($class); } if (!empty($params)) { // Create an instance using Reflection, because constructor has parameters $class = new ReflectionClass($proxyClassName); $instance = $class->newInstanceArgs($params); } else { $instance = new $proxyClassName(); } return $instance; } </class> |
Example:
1 2 3 | $oClass = getProxy( 'myClass', array( 'some params')); $oClass->protected_myProtectedFcuntion(); ... |
Tales from the eXtreme Side - IPC 07 is over
So finally after 5 days I survived IPC and made my way home. As always it was a real pleasure to meet “la famiglia” - I learned a lot, hopefully could teach other people something and definitly had too much beer. The real reason why the IPC is so short is probably that otherwise 90% of PHP community would have to detox from alcohol
So - the title from my last talk could have also been the title of the whole IPC.
For those of you who are interested in the slides - you can download them directly here: tales-from-the-extreme-side_ipc_07.pdf
or if you interested in the PPT you might be able to fetch them via http://phpconference.com/
Using safari on MAC OS X using selenium
Hoi,
while setting up our new testing Machine ( a very old PowerPC with 466 Mhz ) I realized that it is not that easy to make Safari work using Selenium RC. With Michele from mayflower helping me we could make it work.
Here is the solution:
1. Use this patched Version of Selenium RC Snapshot with Safari and MAC OS X support ( it is the snapshot including the patch from here ).
2. Start Selenium using sudo
3. If you face any problems like “Please add the directory containing networksetup to your path” then you should check your disc under /System/Library/CoreServices/RemoteManagement/ARDAgent.app/
Contents/Support/networksetup - this directory should exist. If not - like on my old machine - you need to create a symlink to the correct directory - e.g.
“sudo ln -s networksetup-panther networksetup”
That’s it - I hope it helps I needed quite a few hours to figure this out.